MAP496A Recovery actions for special PCIe-related I/O enclosure errors (Models 98x)

This MAP lists SRCs that require special repair actions to be completed by the service representative or the next level of support.

MAP496A Section-1

Procedure

  1. Select the appropriate rack model.
  2. Does the FRU list, in the serviceable event, that sent you here contain a symbolic FRU similar to Invalid-MTMS-cpssebay**?
  3. When the FRU list contains a symbolic FRU similar to Invalid-MTMS-cpssebay**, the location code is invalid and cannot be used to determine the failing I/O enclosure.
  4. To determine the cpssebay** value from the symbolic FRU of Invalid-MTMS-cpssebay**, use the first column of Table 1, Table 2, or Table 3.
    Table 1. Symbolic FRU location to type-location translation (Models 981, 982, 985, 986, 988)
    Symbolic FRU location code Location Type location I/O enclosure number
    cpssebay00 1B1 1500-1B1 0
    cpssebay01 1B2 1500-1B2 1
    cpssebay02 1B3 1500-1B3 2
    cpssebay03 1B4 1500-1B4 3
    cpssebay04 2B1 1500-2B1 4
    cpssebay05 2B2 1500-2B2 5
    cpssebay06 2B3 1500-2B3 6
    cpssebay07 2B4 1500-2B4 7
    Table 2. Symbolic FRU location to type-location translation (Models 980, 984)
    Symbolic FRU location code Location Type-location I/O enclosure number
    cpssebay02 1B3 1500-1B3 2
    cpssebay03 1B4 1500-1B4 3
    cpssebay06 2B3 1500-2B3 6
    cpssebay07 2B4 1500-2B4 7
    Table 3. Symbolic FRU location to type-location translation (Model 984 all-flash)
    Symbolic FRU location code Location Type-location I/O enclosure number
    cpssebay00 1B1 1500-1B1 0
    cpssebay01 1B2 1500-1B2 1
    cpssebay02 1B3 1500-1B3 2
    cpssebay03 1B4 1500-1B4 3
  5. Determine the location code in Table 1, Table 2, or Table 3, second column for the symbolic FRU location code in the FRU list.
  6. Translate the three-character location code from the previous step to a physical location of the I/O enclosure in the rack. See Figure 1.
    Figure 1. I/O enclosure locations (front) (models 98x, 8xE)
    Image that shows the front I/O enclosure locations (models 98x).
    Note: Enclosures 1B1, 1B2, 2B1, and 2B2 are not present in some models.
  7. Update the HMC microcode objects for the I/O enclosure machine/type/model/serial number by using a pseudo repair of the PCIe and PCN card FRU. The update causes the I/O enclosure to be power-cycled.
    1. From the navigation area, click Storage Facility Management > storage facility.
    2. From the Task area, click Exchange Parts > Exchange IO Enclosure and Components.
    3. Click Show I/O Enclosures and select the enclosure location.
    4. Click Show FRUS.
    5. Select I/O Enclosure PCIe and PCN Card and then click Exchange FRU.
    6. When prompted to replace the FRU, do not disconnect the PCIe and PCN cables from the card. Do not remove the card.

      Continue the repair.

    7. If the repair is successful, exit this MAP and ensure that any related serviceable events are closed.
    8. If the repair fails with the same error, replace the I/O enclosure PCIe and PCN card.

MAP496A Section-2

Procedure

  1. Find your SRC in Table 4.
    Table 4. Repair actions for special SRCs
    SRCs that require special repairs
    SRCs Action
    BE1E2197 A CEC to CEC path heartbeat timeout was reported by CEC1 (LPAR ESS11). Go to MAP496A Section-3.
    BE1E2198 A CEC to CEC path heartbeat timeout was reported by CEC0 (LPAR ESS01). Go to MAP496A Section-3.
    BE1E25AA A single CEC to I/O enclosure PCIe link fault was detected during a CEC service action. Go to MAP496A Section-3.
    BE1E25AB Multiple CEC to I/O enclosure PCIe link faults were detected during a CEC service action. Go to MAP496A Section-3.
    BE370012 PCIe I/O enclosure discovery failure (missing I/O enclosure). Go to MAP496A Section-4.
    BE38256B PCIe enclosure discovery/configuration failure. Could not initialize path from local server to I/O enclosure. Go to MAP496A Section-3.
    BE38256C I/O enclosure FPGA update image corrupted on local server. Contact your next level of support.
    BE38256D PCIe I/O enclosure FPGA error. Contact your next level of support.
    BE38256E PCIe I/O enclosure MTMS unknown/invalid. Contact your next level of support.
    BE38256F PCIe I/O enclosure mis-cabling detected. Go to MAP496A Section-3.
    BE382572 Error occurred during I/O enclosure error data collection. Go to MAP496A Section-3.
    BE38257B PCIe interface to PCIe I/O enclosure down. Go to MAP496A Section-3.
    BE382563 Multi-PCIe link degraded detected on the local server. Contact your next level of support.
    BE382566 PCIe I/O enclosure discovery/configuration failure. Go to MAP496A Section-3.
    BE382567 Invalid server config. Contact your next level of support.
    BE382574 One LPAR cannot communicate the I/O enclosure; a system failover is required. Go to MAP496A Section-3.
    BE382575 PCIe I/O enclosure discovery failure (missing an I/O enclosure). Go to MAP496A Section-4.
    Any other SRC Contact your next level of support.
  2. Use the Action column entry to continue the repair.

MAP496A Section-3

About this task

The serviceable event FRU list that sent you here contains one or more cables and possibly more FRUs.
Important: Both ends of each PCIe cable appear in the FRU list. Only the first cable location code is available to select for repair or replace for each cable in the FRU list. The subsequent CBLCONT location code shows where a cable continues to connect to, but is not available to select for repair or replace.

Procedure

  1. Inspect both ends of each PCIe cable listed in the FRU list.
    1. Do not plug or unplug the cable.
    2. Refer to the following cabling diagrams based on the number of installed I/O enclosures in the machine. The CBLCONT location code that is listed is the port on the I/O enclosure where the cable is supposed to be connected.

      Based on the appropriate cable figure, check each end of the cable that is listed on the screen that sent you here to ensure that it is properly plugged into the correct connector.

    3. Observe the body of the cable to ensure that it is not damaged.
    Figure 2. PCIe cables, Models 981, 985, 986, 98E, 85E, 86E (rear view, all eight I/O enclosures shown)
    Image that shows the eight I/O enclosures for Models 981, 985, 986,98E, 85E, 86E
    Figure 3. PCIe cables, Models 980, 984, 98B, 84E (rear view, all four I/O enclosures shown)
    Image that shows the four I/O enclosures for Models 980, 98B
    Figure 4. PCIe cables, Model 982, 988, 98F, 88E
    PCIe cables, Model 982, 988. 98F, 88E
    Figure 5. PCIe cables, Model 984 all-flash
    PCIe cables, Model 984 all-flash
  2. Is the PCIe cable properly plugged and not damaged?
    • Yes, go to the next step.
    • No, go to step 5.
  3. The cable is properly plugged and is not damaged.
    Did you reach this step after replacing both the I/O enclosure PCIe and PCN card and the I/O enclosure backplane?
    • No, go to the next step.
    • Yes, a pseudo-repair of the PCIe and PCN card might recover this condition. Complete the following steps:
      1. Return to the screen that sent you here.
      2. To the question, "What was the result of using the service procedure from Infocenter?" click Problem not fixed and then click Next.
      3. To the question, "Did you exchange any parts,?" click No and then click Next.
      4. To the question, "Did you isolate the problem,"? click Yes and then click Next.
      5. The current repair action ends, but the serviceable event is left open. Use
        the Exchange Parts menu to complete a pseudo-repair of the I/O enclosure PCIe and PCN card:
        • Storage Facility Management > storage facility > Exchange Parts

        Remove I/O enclosure power when instructed to do so in the exchange procedure, but you do not need to uncable or remove the PCIe and PCN card.
  4. The cable is properly plugged and is not damaged.
    The I/O enclosure PCIe and PCN card and the I/O enclosure backplane were both not replaced.
    1. Return to the screen that sent you here.
    2. To the question, "What was the result of using the service procedure from Infocenter?" click Problem not fixed and then click Next.
    3. To the question, "Did you exchange any parts?" click No and then click Next.
    4. To the question, "Did you isolate the problem"? click No and then click Next.
    5. The next FRU in the list is displayed. Continue the repair by replacing the remaining FRUs until the problem is fixed. Exit this MAP.
  5. The cable is incorrectly plugged or damaged. Refer to Table 5. Select the condition that applies and take the action listed.
    Table 5. Incorrectly plugged or damaged cable - actions to take
    Action in progress when this MAP was reached: Action to take:
    MES to install a new I/O enclosure pair Go to step 6.
    MES to upgrade PCIe and PCN cards in an existing I/O enclosure pair Go to step 7.
    I/O enclosure service action Go to step 7.
  6. You identified incorrectly plugged or damaged cables during a failed installation of an I/O enclosure pair.
    1. Exit this repair action.
    2. Obtain replacement cables for any damaged cables.
    3. Retry the original MES installation with the cables properly connected.
  7. You identified incorrectly plugged or damaged cables during a repair or during an upgrade of the I/O enclosure PCIe and PCN cards.
    1. Return to the screen that sent you here.
    2. To the question, "What was the result of using the service procedure from Infocenter?" click Problem not fixed and then click Next.
    3. To the question, "Did you exchange any parts?" click No and then click Next.
    4. To the question, "Did you isolate the problem?" click No and then click Next.
    5. When the next FRU in the list is displayed, pretend that the other FRUs in the previous FRU list are not available onsite to be replaced.
    6. When asked if the FRU is available to be replaced, answer no. This answer causes each FRU in the list to be displayed until the incorrectly plugged cable or the damaged cable is displayed.

      When the incorrectly plugged cable or the damaged cable is displayed, do a normal FRU replace.

    7. When the repair is complete, exit this MAP.

MAP496A Section-4

Procedure

  1. Observe the FRU list in the serviceable event details that sent you here. It should include one or more of the following FRUs:
    • I/O enclosure PCIe and PCN card
    • I/O enclosure backplane
  2. Display open serviceable events that need repair. Is there any other serviceable event with either FRUs determined in step 1 or with other FRUs such as power supply or fan from this I/O enclosure?
    • Yes, exit this MAP and attempt to repair that serviceable event first.

      If that repair does not correct this problem, return here and continue with the next step.

      If that repair does correct this problem, remember to also close this serviceable event.

    • No, go to the next step.
  3. Inspect both ends of both PCIe cables that are associated with the I/O enclosure listed in the FRU list, that is, intended to be connected to this I/O enclosure.
    1. Do not plug or unplug the cables.
    2. Refer to Figure 2, Figure 3, Figure 4, or Figure 5 for your machine type and model. Based on the appropriate cable figure, check each end of both cables that are intended to be connected to this I/O enclosure to see whether they are properly plugged into the correct connector.
    3. Observe the body of the cable to ensure that it is not damaged.
  4. Are the PCIe cables to the I/O enclosure properly plugged and not damaged?
    • Yes, go to the next step.
    • No, go to step 6.
  5. You have reached this step because the cables are properly plugged and are not damaged.
    1. Return to the screen that sent you here.
    2. To the question, "What was the result of using the service procedure from Infocenter?" click Problem not fixed and then click Next.
    3. To the question, "Did you exchange any parts?" click No and then click Next.
    4. To the question, "Did you isolate the problem?" click No and then click Next.
    5. The next FRU in the list is displayed. Continue the repair by replacing the remaining FRUs until the problem is fixed.

      Exit this MAP.

  6. At least one cable is incorrectly plugged or damaged. Refer to Table 6. Select the condition that applies and take the action listed.
    Table 6. Incorrectly plugged or damaged cable - actions to take
    Action in progress when this MAP was reached: Action to take:
    MES to install a new I/O enclosure pair Go to step 7.
    MES to upgrade PCIe and PCN cards in an existing I/O enclosure pair Go to step 8.
    I/O enclosure service action Go to step 8.
  7. You identified incorrectly plugged or damaged cables during a failed installation of an I/O enclosure pair.
    1. Exit this repair action.
    2. Obtain replacement cables for any damaged cables.
    3. Retry the original MES installation with the cables properly connected.
  8. You identified incorrectly plugged or damaged cables during a repair or during an upgrade of the I/O enclosure PCIe and PCN cards.
    1. Return to the screen that sent you here.
    2. To the question, "What was the result of using the service procedure from Infocenter?" click Problem not fixed and then click Next.
    3. To the question, "Did you exchange any parts?" click No and then click Next.
    4. To the question, "Did you isolate the problem?" click No and then click Next.
    5. The next FRU in the list is displayed. Continue the repair on this FRU, but when instructed to replace the FRU, do not replace that FRU, but instead replace the damaged cables that are connected to the I/O enclosure.
    6. If the repair completes successfully, exit this MAP. Otherwise, contact your next level of support.