From sacadmin Thu May 29 12:20:18 2008 Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m4TJKILb013177; Thu, 29 May 2008 12:20:18 -0700 (PDT) Received: (from ehring@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id m4TJKHGa013173; Thu, 29 May 2008 12:20:18 -0700 (PDT) Date: Thu, 29 May 2008 12:20:18 -0700 (PDT) From: Stephen Ehring Message-Id: <200805291920.m4TJKHGa013173@sac.sfbay.sun.com> To: FWARC-record@sac.sfbay.sun.com Subject: New MD Nodes to Support I/O Reconfiguration for Failover Conditions. [FWARC/2008/349 FastTrack timeout 06/05/2008] Status: RO Content-Length: 604 Template Version: @(#)sac_nextcase 1.66 04/17/08 SMI This information is Copyright 2008 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: New MD Nodes to Support I/O Reconfiguration for Failover Conditions. 1.2. Name of Document Author/Supplier: Author: Marc Fidler 1.3 Date of This Document: 29 May, 2008 4. Technical Description See the case directory for more detail 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: unknown 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From sacadmin Thu May 29 12:26:21 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m4TJQKwK013322 for ; Thu, 29 May 2008 12:26:21 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m4TJQHHG011534 for <@sunmail2sca.sfbay.sun.com:fwarc@sun.com>; Fri, 30 May 2008 03:26:19 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K1N00E0H9ZT6X00@nwk-avmta-1.sfbay.Sun.COM> for fwarc@sun.com (ORCPT fwarc@sun.com); Thu, 29 May 2008 12:26:17 -0700 (PDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K1N00D0E9ZT7G10@nwk-avmta-1.sfbay.Sun.COM> for fwarc@sun.com (ORCPT fwarc@sun.com); Thu, 29 May 2008 12:26:17 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m4TJQGmU021374 for ; Thu, 29 May 2008 19:26:16 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K1N00J019PQDP00@mail-amer.sun.com> (original mail from Stephen.Ehring@Sun.COM) for fwarc@sun.com (ORCPT fwarc@sun.com); Thu, 29 May 2008 13:26:16 -0600 (MDT) Received: from ehring-macbook-pro.local ([129.150.38.78]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K1N002H89ZKJ510@mail-amer.sun.com>; Thu, 29 May 2008 13:26:09 -0600 (MDT) Date: Thu, 29 May 2008 15:26:07 -0400 From: Stephen Ehring Subject: FWARC 2008/349: New MD Nodes to Support I/O Reconfiguration for Failover Conditions. Sender: Stephen.Ehring@sun.com To: fwarc@sun.com, John Johnson Message-id: <483F034F.5020505@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) Status: RO Content-Length: 5183 I am sponsoring this case for Marc Fidler and John Johnson. The case defines new MD nodes to associate different IO topology paths to the upstream ports of PCIE switches. This case is attempting to solve a long standing problem where I/O topology changes necessitate an operating system reinstall. The interfaces in this case can be released in a micro/minor version of the OS and firmware. The case is set to expire on 6/5/2008 http://sac.eng/Archives/CaseLog/arc/FWARC/2008/349/onepager.txt Steve 1. Introduction 1.1. Project/Component Working Name: New MD Nodes to Support I/O Reconfiguration for Failover Conditions. 1.2. Name of Document Author/Supplier: Marc Fidler 1.3. Date of This Document: 05/27/2008 1.4. Email Aliases: 1.4.1. Responsible Manager: Chad Solomon 1.4.2. Responsible Engineer: Marc Fidler 1.4.3. Interest List: John Johnson 2. Project Summary 2.1. Project Description: This project defines new MD nodes to represent the current and potential paths to a PCIe switch when multiple IO topology configurations are possible. 3. Business Summary 3.1. Problem Area: Current architecture prevents operating system boot if the PCIE device paths have changed since the previous OS boot. This can happen, for example, due to a host-to-PCIE bridge being eliminated from the system where the PCI switches are re-routed to work around the missing bridge. This case is attempting to solve a long standing problem where I/O topology changes necessitate an operating system reinstall. 4. Technical Description: 4.1. Details: This project defines new MD nodes to associate all possible device paths to the upstream port of a PCIE switch. The operating system will use these nodes to remap existing device node paths. Solaris uses the device node paths to configure devices. A unique device_instance number is created in /dev and /etc/path_to_inst file for every I/O path. Solaris expects the I/O path to be constant. When the same device is found at a different I/O patch after power-cycle it is treated as a new device, which can happen when a node is removed or added. In order for vBSC to remain stateless, there must be some mechanism to name all of the potential PCIe paths to a switch. For example, on Batoka, the switch below cpu0 can be reached with the following paths: /pci@400 /pci@500/pci@0/pci@1/pci@0/pci@1 /pci@600/pci@0/pci@1 /pci@700/pci@0/pci@1/pci@0/pci@1/pci@0/pci@1 When cpu0 exists, we use the first path (/pci@400) to access the switch, and the rest are aliases. So the resulting ioalias node would be: node ioalias pcie0 { current = "/pci@400"; aliases = "/pci@500/pci@0/pci@1/pci@0/pci@1" + "/pci@600/pci@0/pci@1" + "/pci@700/pci@0/pci@1/pci@0/pci@1/pci@0/pci@1"; back -> ioaliases; }; Software has developed a script that looks through the alias list, and check for device paths below the aliases paths. Any that exist will be re-written to use the current path. 4.1.1 ioalias device node Name: ioalias Category: optionally required by ioaliases Required Subordinates: Optional Subordinates: Description: This node provides the current path to a PCIe switch's upstream port and a list of all possible paths to the same PCIe switch upstream port. Properties: Name Tag Req'd? Description ---------------------------------------------------------------------- current PROP_STR yes A string containing the path to a PCIe switch upstream port. aliases PROP_STR yes A list of space separated strings containing all the possible paths to a PCI root nexus. 4.1.2 ioaliases device node Name: ioaliases Category: optionally required by root Required Subordinates: Optional Subordinates: ioalias device node Description: This node provides forward pointers to all system ioalias nodes For example on Batoka system the ioaliases node would be: node ioaliases ioaliases { fwd -> pcie0; fwd -> pcie1; fwd -> pcie2; fwd -> pcie3; }; 4.2 Interface tables 4.2.1 Imported Interfaces Name Classification Description ----------------- --------------- ------------------- sun4v Machine Sun Private MD nodes definitions as Description nodes defined by FWARC/2005/115 4.5.2 Exported Interfaces Name Classification Description ---------------------- -------------- ------------------- ioaliases and ioalias Sun Private MD nodes defined MD nodes by this case From sacadmin Fri Jun 6 00:07:52 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5677qS2012934 for ; Fri, 6 Jun 2008 00:07:52 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m5677oGo024422 for <@sunmail2sca.sfbay.sun.com:fwarc@sun.com>; Fri, 6 Jun 2008 08:07:51 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K2100D01552S500@nwk-avmta-1.sfbay.Sun.COM> for fwarc@sun.com (ORCPT fwarc@sun.com); Fri, 06 Jun 2008 00:07:50 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K210092I552VHD0@nwk-avmta-1.sfbay.Sun.COM> for fwarc@sun.com (ORCPT fwarc@sun.com); Fri, 06 Jun 2008 00:07:50 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m5677nbd028630 for ; Fri, 06 Jun 2008 00:07:49 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K21002014Y98L00@fe-sfbay-10.sun.com> (original mail from Hitendra.Zhangada@Sun.COM) for fwarc@sun.com (ORCPT fwarc@sun.com); Fri, 06 Jun 2008 00:07:49 -0700 (PDT) Received: from [129.150.32.151] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K2100MGO5506UA0@fe-sfbay-10.sun.com>; Fri, 06 Jun 2008 00:07:49 -0700 (PDT) Date: Fri, 06 Jun 2008 00:07:50 -0700 From: Hitendra Zhangada Subject: Re: FWARC 2008/349: New MD Nodes to Support I/O Reconfiguration for Failover Conditions. In-reply-to: <483F034F.5020505@sun.com> Sender: Hitendra.Zhangada@sun.com To: fwarc@sun.com Cc: John Johnson , Marc Fidler Message-id: <4848E246.6060601@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <483F034F.5020505@sun.com> User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) Status: RO Content-Length: 2288 Stephen Ehring wrote: I know timer has already expired on this but I do have couple of comments for clarification. When the boot device path changes, OpenBoot will create and boot from a different device path than used prior to the change. The proposed MD nodes will enable a way for Solaris to detect "aliases" and if found one then it will allow boot from the new device path. This is what this case solves, right? > > 4.1.1 ioalias device node I think it should be MD node, not device node. > Name: ioalias > Category: optionally required by ioaliases > Required Subordinates: > Optional Subordinates: > Description: This node provides the current path to a > PCIe > switch's upstream port and a list of all possible paths > to the same PCIe switch upstream port. > Properties: > Name Tag Req'd? Description > > ---------------------------------------------------------------------- > current PROP_STR yes A string containing the path to a > PCIe switch upstream port. > aliases PROP_STR yes A list of space separated > strings containing all the > possible paths to a PCI root nexus. Does empty or NULL string represent no aliases? I would think so but want to be clear. > > 4.1.2 ioaliases device node Same as above, MD node not device node. > Name: ioaliases > Category: optionally required by root > Required Subordinates: > Optional Subordinates: ioalias device node > Description: This node provides forward pointers to all > system ioalias nodes > > For example on Batoka system the ioaliases node would be: > node ioaliases ioaliases { > fwd -> pcie0; > fwd -> pcie1; > fwd -> pcie2; > fwd -> pcie3; > }; > Thanks. -- Hitendra Zhangada ============================================= SPS Common SW Features Engineering Software Group, Sun Microsystems, Inc. Work Ph# (858) 625 3757, Ext. x53757 SUN Internal homepage http://esp.west/~hitu From sacadmin Fri Jun 6 02:05:19 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5695JtS015253 for ; Fri, 6 Jun 2008 02:05:19 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m5695HZU025887 for <@sunmail2sca.sfbay.sun.com:fwarc@sun.com>; Fri, 6 Jun 2008 02:05:19 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K210020HAKUO400@brm-avmta-1.central.sun.com> for fwarc@sun.com (ORCPT fwarc@sun.com); Fri, 06 Jun 2008 03:05:18 -0600 (MDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K21000AQAKU9GB0@brm-avmta-1.central.sun.com> for fwarc@sun.com (ORCPT fwarc@sun.com); Fri, 06 Jun 2008 03:05:18 -0600 (MDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m5695IGL014857 for ; Fri, 06 Jun 2008 09:05:18 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K2100L01AK56500@mail-amer.sun.com> (original mail from Marc.Fidler@Sun.COM) for fwarc@sun.com (ORCPT fwarc@sun.com); Fri, 06 Jun 2008 03:05:18 -0600 (MDT) Received: from [192.168.1.3] ([76.119.250.234]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K21006Y0AKTGJ60@mail-amer.sun.com>; Fri, 06 Jun 2008 03:05:17 -0600 (MDT) Date: Fri, 06 Jun 2008 05:05:14 -0400 From: Marc Fidler Subject: Re: FWARC 2008/349: New MD Nodes to Support I/O Reconfiguration for Failover Conditions. In-reply-to: <4848E246.6060601@sun.com> Sender: Marc.Fidler@Sun.COM To: Hitendra Zhangada Cc: fwarc@Sun.COM, John Johnson Message-id: <4848FDCA.90601@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <483F034F.5020505@sun.com> <4848E246.6060601@sun.com> User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) Status: RO Content-Length: 2325 Hitendra, Please see comments below. Hitendra Zhangada wrote: > Stephen Ehring wrote: > > > I know timer has already expired on this but I do have couple of comments > for clarification. > > When the boot device path changes, OpenBoot will create and boot from > a different device path than used prior to the change. The proposed > MD nodes > will enable a way for Solaris to detect "aliases" and if found one > then it will allow > boot from the new device path. This is what this case solves, right? That is correct. > > >> >> 4.1.1 ioalias device node > > I think it should be MD node, not device node. > >> Name: ioalias >> Category: optionally required by ioaliases >> Required Subordinates: >> Optional Subordinates: >> Description: This node provides the current path to a >> PCIe >> switch's upstream port and a list of all possible paths >> to the same PCIe switch upstream port. >> Properties: >> Name Tag Req'd? Description >> >> ---------------------------------------------------------------------- >> current PROP_STR yes A string containing the path >> to a >> PCIe switch upstream port. >> aliases PROP_STR yes A list of space separated >> strings containing all the >> possible paths to a PCI root nexus. > > Does empty or NULL string represent no aliases? I would think so but > want to be clear. Correct. Marc > >> >> 4.1.2 ioaliases device node > > Same as above, MD node not device node. > >> Name: ioaliases >> Category: optionally required by root >> Required Subordinates: >> Optional Subordinates: ioalias device node >> Description: This node provides forward pointers to all >> system ioalias nodes >> >> For example on Batoka system the ioaliases node would be: >> node ioaliases ioaliases { >> fwd -> pcie0; >> fwd -> pcie1; >> fwd -> pcie2; >> fwd -> pcie3; >> }; >> > > > Thanks. > -- Marc Fidler Sun Microsystems, Inc. (781) 442-3445 From sacadmin Wed Jun 11 12:22:08 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5BJM71r025887 for ; Wed, 11 Jun 2008 12:22:08 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m5BJLu5w004954 for <@sunmail2sca.sfbay.sun.com:fwarc@sun.com>; Thu, 12 Jun 2008 03:22:06 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K2B00415CGSUC00@brm-avmta-1.central.sun.com> for fwarc@sun.com (ORCPT fwarc@sun.com); Wed, 11 Jun 2008 13:22:04 -0600 (MDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K2B000J9CGSIZ30@brm-avmta-1.central.sun.com> for fwarc@sun.com (ORCPT fwarc@sun.com); Wed, 11 Jun 2008 13:22:04 -0600 (MDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m5BJM4ST016827 for ; Wed, 11 Jun 2008 19:22:04 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K2B00201B88QR00@mail-amer.sun.com> (original mail from Stephen.Ehring@Sun.COM) for fwarc@sun.com (ORCPT fwarc@sun.com); Wed, 11 Jun 2008 13:22:04 -0600 (MDT) Received: from [129.148.131.32] by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K2B00KEZCGAF5B0@mail-amer.sun.com>; Wed, 11 Jun 2008 13:21:51 -0600 (MDT) Date: Wed, 11 Jun 2008 15:21:46 -0400 From: Stephen Ehring Subject: Re: FWARC 2008/349: New MD Nodes to Support I/O Reconfiguration for Failover Conditions. In-reply-to: <4848E246.6060601@sun.com> Sender: Stephen.Ehring@sun.com To: Hitendra Zhangada Cc: fwarc@sun.com, John Johnson , Marc Fidler Reply-to: Stephen.Ehring@sun.com Message-id: <485025CA.3070802@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <483F034F.5020505@sun.com> <4848E246.6060601@sun.com> User-Agent: Thunderbird 2.0.0.15pre (X11/20080608) Status: RO Content-Length: 542 The timer on this case has expired, so it is closed as approved. The interfaces can be released in a micro/minor version of the firmware. Steve FYI, there were two minor cosmetic changes to the onepager that went out; the nodes were referred to as device nodes and not MD nodes, and the example node was missing the back link to root: 67c67 < 4.1.1 ioalias device node --- > 4.1.1 ioalias MD node 83c83 < 4.1.2 ioaliases device node --- > 4.1.2 ioaliases MD node 96a97 > back -> root;