1 Introduction The sun4v delivers interrupt notifications to a virtual cpu by means of a "device mondo" queue (see [2]). Two registers are used to hold the head and tail pointers to this round-robin queue, and the hardware automatically delivers an interrupting trap to a virtual cpu when the head != tail and interrupts are enabled. Each interrupt entry in the device mondo queue is a fixed 64 Bytes in size and is used to hold a modest amount of additional information regarding the interrupt it represents. The first 64-bit word of each dev-mondo packet holds an identifier for the interrupt source, and the remaining 7 words are defined to be interrupt source specific. 1.1 Legacy use The initial Niagara (T1) hypervisor supplies its sysino in word 0 of each dev-mondo. The hypervisor's sysino is derived from the actual device handle and ino of the interrupt source using the Hypervisor's INTR_DEVINO2SYSINO API call. The sysino API was intended for the Hypervisor to return any 64-bit value to represent an interrupt source. The arbitrary sysino value was intended such that any algorithm might be employed in generating a sysino for the corresponding device handle and interrupt number. In practice the implementation was simply to concatenate the devhandle and ino values into a single 64-bit sysino number. Solaris 10 uses this sysino value as an index into a linear table programmed with information relevent to the specific interrupt source. The size of this table fixed at Solaris compile time as a function of the number of cpus. The above assumption made by Solaris requires that the sysinos supplied in each dev-mondo lie in the range 0-2047 - the size of the table when Solaris is compiled for 64 cpus. There is no mechanism to enforce this contract between guest OS and the Hypervisor. Sysinos generated by the hypervisor that are out of range of the table are silently dropped (interrupts are lost), and worse, the upper end of the Solaris table is used for software induced timer interrupts, so unfortunate generation of Hypervisor sysinos can in fact be interpreted as interrupts other than those for the device they represent. The additional hurdle of dynamic assignment of sysinos presents itself for Logical Domaining and Live Migration. Both features require the ability to dynamically assign and delete interrupt sources for a guest OS, and furthermore transfer those assignments between machines. 1.2 Interrupt cookies This document describes an extension to the existing Hypervisor interrupt infrastructure to solve the aforementioned problems. Guest OSs and Hypervisor will be transitioned over to the new mechanism. Instead of a sysino to identify an interrupt source, a guest OS will set a cookie value of its choice. This cookie is returned as word 0 in a dev-mondo entry when the interrupt occurs. The cookie may be defined and interpreted in anyway by the guest - for example as a pointer to an internal data structure for the interrupt. Though legacy interrupt sources (for example the existing PCI-E infrastructure on Ontario/Erie) may have cookie support in the Hypervisor, the corresponding guest OS nexus drivers must continue to provide support for existing hypervisor defined sysinos so as to continue to function on legacy firmware implementations. Similarly, new firmware implementations should countinue to provide support for sysino based interrupt APIs, in order to support legacy guest OS nexus drivers. Section 3 of this document defines the APIs used to set and get interrupt cookies in addition to APIs to manipulate the interrupt state machine using by dev_handle and ino - thus removing the need for the sysino and the problems of its dynamic allocation and migration between machines. 2 References [1] Sun4v Hypervisor API http://projectq.sfbay.sun.com/docs/api.pdf [2] Sun4v Architecture Specification http://projectq.sfbay.sun.com/docs/sun4v.pdf [3] Hyperprivileged Reference Architecture http://projectq.sfbay.sun.com/docs/hyperpriv.pdf 3 Description The interrupt APIs (0xa0 through 0xa6) are currently available to a guest when it negotiates a version in the core API group 0x1. This project will create a new interrupt group numbered 0x2 and move these functions to this group. These API functions will be available to a guest when it negotiates version 1.0 in this group. The list of APIs being migrated to group 0x2 are listed below: INTR_DEVINO2SYSINO 0xa0 INTR_GETENABLED 0xa1 INTR_SETENABLED 0xa2 INTR_GETSTATE 0xa3 INTR_SETSTATE 0xa4 INTR_GETTARGET 0xa5 INTR_SETTARGET 0xa6 This project will not change the actual behavior of these APIs and they will continue to function as described in the case FWARC/2005/116. A guest has to now negotiate version 1.0 in group 0x2 prior to accessing these APIs. This project will also deliver the new interrupt APIs specified below, that allow a guest to specify a single 64-bit cookie that will be delivered in the first word (word 0) of a dev_mondo packet. It also provides modified interrupt APIs that use the devhandle and devino to refer to the interrupt source instead of the sysino provided by the Hypervisor via the INTR_DEVINO2SYSINO API. The new interrupt API functions will be available to a guest when it negotiates version 2.0 in the interrupt API group 0x2. When a guest negotiates v2.0, all interrupt sources will only support using the cookie interface, and any attempt to use the version 1.0 INTR_xxx APIs numbered 0xa0 to 0xa6 will result in ENOTSUPPORTED being returned. Interrupts from all sources are explicitly disabled until the guest that negotiated v2.0 in group 0x2, sets a valid cookie value for the interrupt source. A guest may upgrade to using the cookie based interrupt APIs, by negotiating version 2.0 in group 0x2, even if it had previously negotiated version 1.0 in group 0x2. Subsequent accesses to v1.0 interrupt APIs in group 0x2 will fail with ENOTSUPPORTED. Two different guests running in a system can negotiate different versions in API group 0x2, but a single guest can negotiate either version 1.0 or 2.0 in group 0x2 and use the corresponding APIs. 3.1 Interrupt cookie API Function Number Definitions VINTR_GETCOOKIE 0xa7 VINTR_SETCOOKIE 0xa8 VINTR_GETENABLED 0xa9 VINTR_SETENABLED 0xaa VINTR_GETSTATE 0xab VINTR_SETSTATE 0xac VINTR_GETTARGET 0xad VINTR_SETTARGET 0xae 3.2 Virtual interrupt API Definitions 3.2.1 vintr_getcookie trap #FAST_TRAP function# VINTR_GETCOOKIE arg0 devhandle arg1 devino ret0 status ret1 cookie_value Gets the cookie values that will be delivered in word 0 of a dev_mondo packet to a guest. In the event that no cookie has been set, the value of 0 is returned. 3.2.1.1 Errors EINVAL Invalid devhandle or devino ENOTSUPPORTED (Virtual) device does not support cookies 3.2.2 vintr_setcookie trap #FAST_TRAP function# VINTR_SETCOOKIE arg0 devhandle arg1 devino arg2 cookie_value ret0 status Sets the cookie value that will be delivered in word 0 of a dev_mondo packet to a guest. A call to this API will overwrite any previous cookie values set via the same API. If cookie_value is 0 the interrupt source is returned to the state of having no cookie assigned, and interrupts are explicity disabled for the device. 3.2.2.1 Errors EINVAL Invalid devhandle or devino, or cookie_value is in range 1 to 2047 ENOTSUPPORTED (Virtual) device does not support cookies EWOULDBLOCK Operation would block 3.2.3 vintr_getenabled trap #FAST_TRAP function# VINTR_GETENABLED arg0 devhandle arg1 devino ret0 status ret1 intr_enabled Returns state in intr_enabled for the interrupt defined by devino. Return values are: INTR_ENABLED or INTR_DISABLED 3.2.3.1 Errors EINVAL Invalid devhandle or devino ENOTSUPPORTED (Virtual) device does not support the interface 3.2.4 vintr_setenabled trap #FAST_TRAP function# VINTR_SETENABLED arg0 devhandle arg1 devino arg2 intr_enabled ret0 status Sets the 'enabled' state of the interrupt sysino legal values for intr_enabled are: INTR_ENABLED or INTR_DISABLED 3.2.4.1 Errors EINVAL Invalid devhandle or devino ENOTSUPPORTED (Virtual) device does not support the interface 3.2.5 vintr_getstate trap #FAST_TRAP function# VINTR_GETSTATE arg0 devhandle arg1 devino ret0 status ret1 intr_state Returns the current state of the interrupt given by the devino argument. 3.2.5.1 Errors EINVAL Invalid devhandle or devino ENOTSUPPORTED (Virtual) device does not support the interface 3.2.6 vintr_setstate trap #FAST_TRAP function# VINTR_SETSTATE arg0 devhandle arg1 devino arg2 intr_state ret0 status Sets the current state of the interrupt given by the devino argument to the value given in the argument intr_state. Note: Setting the state to INTR_IDLE clears any pending interrupt for devino. 3.2.6.1 Errors EINVAL Invalid devhandle or devino ENOTSUPPORTED (Virtual) device does not support the interface 3.2.7 vintr_gettarget trap #FAST_TRAP function# VINTR_GETTARGET arg0 devhandle arg1 devino ret0 status ret1 cpuid Returns the cpuid that is the current target of the interrupt given by the devino argument. The cpuid value returned is undefined if the target has not been set via vintr_settarget. 3.2.7.1 Errors EINVAL Invalid devhandle or devino ENOTSUPPORTED (Virtual) device does not support the interface 3.2.8 vintr_settarget trap #FAST_TRAP function# VINTR_SETTARGET arg0 devhandle arg1 devino arg2 cpuid ret0 status Set the target cpu for the interrupt defined by the argument devino to the target cpu value defined by the argument cpuid. 3.2.8.1 Errors EINVAL Invalid devhandle or devino ENOCPU Invalid cpuid ENOTSUPPORTED (Virtual) device does not support the interface