MAP2220 CEC enclosure to RPC card I2C communication failure

A view of the detail data for one of the RPCs interrupt registers is not available from one or both clusters.

About this task

Only the base LPAR in a CEC enclosure communicates with the RPC cards. The LPAR communicates with the service processor through the RTAS interface. The service processor communicates with both RPC cards through a shared I2C interface. The same I2C interface also communicates with the CEC enclosure control panel.

MAP2220 Section-1

Procedure

Display open serviceable events. Find those with the SRCs listed in Table 1. Use Table 1 to prioritize the order of repair.
Table 1. SRC and associated actions
SRC repair order SRC definition1, 2 Go to
BE190010 CEC enclosure 1 failed to communicate with RPC1 and RPC2. MAP2220 Section-2 (One CEC communication failure with both RPC cards)
BE190011 CEC enclosure 2 failed to communicate with RPC1 and RPC2.
BE190012 CEC enclosure 1 and 2 failed to communicate with RPC1. MAP2220 Section-3 (Both CECs communication failure with one RPC card)
BE190013 CEC enclosure 1 and 2 failed to communicate with RPC2.
BE190014 CEC enclosure 1 failed to communicate with RPC1. MAP2220 Section-6 (One CEC communication failure with one RPC card)
BE190015 CEC enclosure 2 failed to communicate with RPC1.
BE190016 CEC enclosure 1 failed to communicate with RPC2.
BE190017 CEC enclosure 2 failed to communicate with RPC2.

Note 1: CEC enclosure 1 is upper, CEC enclosure 2 is lower.

Note 2: For RPC locations, see Figure 1, Figure 2, Figure 3, or Figure 4.
Figure 1. Location codes for the RPC cards (Models 941, 951)
RPC card
Figure 2. Location codes for the RPC cards (Model 961)
Location codes for the RPC cards (Model 961)
Figure 3. Location codes for the RPC card (Model 98x)
Location codes for the RPC card (Model 98x)
Figure 4. Location codes for the RPC card (Model 983), rear of management enclosure
RPC card rear connectors J2xx
Figure 5. Location codes for the CEC enclosure (rear view) (Models 941, 951)
Location codes for the CEC enclosure (rear view)
Figure 6. Location codes for the CEC enclosure (rear view) (Model 961)
Location codes for the CEC enclosure (rear view) (Models 961)
Figure 7. Location codes for the CEC enclosure (rear view) (Models 980, 981, 984, 985, and 986)
Note: Models 981, 985, and 986 CEC are shown; model 980, 983, 984 card locations are similar.
CEC enclosure location codes (rear view)
Figure 8. CEC enclosure location codes (rear view) (Model 982 and 988)
CEC enclosure location codes (rear view) (Model 982)
Figure 9. Location codes for the PPS sequencer module
Location codes for the PPS sequencer module
Figure 10. Location codes for the DC-UPS (Models 961, 96E, 98x three-phase) (rear)
Location codes for the DC-UPS (Models 961, 96E) (rear)
Figure 11. Location codes for the DC-UPS (Models 98x, single-phase) (rear)
Location codes for the DC-UPS (Models 98x) (rear)

MAP2220 Section-2 (One CEC communication failure with both RPC cards)

About this task

One CEC enclosure has a communication failure through its service processor to both RPC cards.

Procedure

  1. Use the SRC in the serviceable event that sent you here and refer to Table 2 for an overall description of your failure. The failure could be caused by one of the following FRUs, which are listed in alphabetical order:
    • Models 941, 951:
      • CEC enclosure control panel
      • CEC enclosure service processor
      • CEC enclosure I/O backplane assembly
      • RPC card
      • RPC card to CEC enclosure cable (I2C Y-Cable)
    • Model 961:
      • CEC enclosure control panel
      • CEC enclosure system backplane (Model 961) (contains service processor function)
      • CEC enclosure I2C card
      • RPC card to CEC enclosure I2C cable
      • RPC card
    • Models 980, 981, 983, 984, 985, 986:
      • CEC enclosure control panel
      • CEC enclosure system backplane (contains service processor function)
      • CEC enclosure I2C card
      • RPC card to CEC enclosure I2C cable
      • RPC card
    • Models 982, 988:
      • CEC enclosure control panel
      • CEC enclosure service processor card
      • CEC enclosure I2C card
      • RPC card to CEC enclosure I2C cable
      • RPC card
    Table 2. SRC communication-failure resources
    SRC Communication failure resources
    BE190010 CEC enclosure 1 failure to communicate with RPC1 card and RPC2 card
    BE190011 CEC enclosure 2 failure to communicate with RPC1 card and RPC2 card
  2. Display open serviceable events that need repair. Is there any other serviceable event with FRUs listed in step 1?
    • Yes, exit this MAP and attempt to repair that serviceable event first. If that repair does not correct this problem, return here and continue with the next step. If that repair does correct this problem, remember to also close this serviceable event.
    • No, go to the next step.
  3. Observe the CEC enclosure control panel. Does the control panel display a ball icon slowly moving clockwise around the display?

MAP2220 Section-3 (Both CECs communication failure with one RPC card)

About this task

Both CEC enclosures have a communication failure through their service processor to the same RPC card. The working RPC card is not reporting any problem with the suspect RPC card. There should be no single point of failure that can cause this failure condition.

Procedure

  1. Use the SRC in the serviceable event that sent you here to determine which RPC card the CEC failed to communicate with. Refer to Table 3.
    Table 3. SRC communication interface-failure resources
    SRC Communication failure resources
    BE190012 Both CEC enclosure service processors, failure to communicate with RPC1 card
    • Models 941, 951, see Figure 1 (RPC1 is on the right, viewed from rear).
    • Model 961 see Figure 2 (RPC1 is the upper card).
    • Model 98x (not 983) see Figure 3. (RPC1 is on the left, viewed from rear)
    • Model 983 see Figure 4. (RPC1 is on the left, viewed from rear)
    BE190013 Both CEC enclosure service processors, failure to communicate with RPC2 card
    • Models 941, 951, see Figure 1 (RPC2 is on the left, viewed from rear).
    • Model 961 see Figure 2 (RPC2 is the lower card).
    • Model 98x (not 983) see Figure 3. (RPC2 is on the right, viewed from rear)
    • Model 983 see Figure 4. (RPC2 is on the right, viewed from rear)
  2. Display serviceable events that need repair. Is there any other serviceable event listing the RPC card determined in step 1 or CEC enclosure to RPC card cable that connects to it?
    • Yes, exit this MAP and attempt to repair that serviceable event first. If that repair does not correct this problem also, return here and continue at the next step. If that repair does not correct this problem, remember to also close this serviceable event.
    • No, go to the next step.
  3. At the suspect RPC card determined in step 1, ensure all the cables are properly connected before you continue. If any cables are not connected, do not connect them now. Instead go to the next step and do a psuedo repair of the RPC card. When you are directed to replace the RPC card, do not replace it but instead connect the cables.
  4. The possible failing FRUs are the RPC card and CEC enclosure to RPC card cable.
    1. Use the Exchange Parts procedure to select the suspect RPC card determined in step 1:
      1. From the navigation area, click Storage Facility Management > storage facility.
      2. From the bottom Task area, click Exchange Parts > Exchange Rack Components .... The Show Rack Enclosures window opens.
      3. Select a rack and click Show FRUs. The Show Rack FRUs window opens.
    2. Select the suspect RPC Card FRU and continue the guided repair.
      1. From the Show Rack FRUs window, select RPC Card and click Exchange FRU.

MAP2220 Section-4

Procedure

  1. The CEC enclosure control panel displays this clockwise moving icon when:
    • The control panel was installed into the CEC enclosure and the service processor firmware has not logically installed the control panel, see step 2.
    • The control panel is failing and needs to be replaced.
  2. The default setting for the control panel is "installed." If the setting is "not installed" and the control panel is installed in the CEC enclosure with CEC power on, the icon moves slowly clockwise around the display. Use the ASMI Concurrent Maintenance > Control Panel menu option to ensure the control panel is logically installed. Refer to the MAP1221 ASMI menu structure.
  3. If the control panel itself is failing, replace the control panel. Refer to MAP1215 Replace a FRU.

MAP2220 Section-5 (One CEC communication failure to both RPC cards, continued)

Procedure

  1. Determine whether the CEC enclosure to RPC card cable is causing the problem.
    • For Models 941, 951 continue at step 2.
    • For Model 961 continue at step 3.
    • For Models 980, 981, 984, 985, 986 continue at step 4.
    • For Models 982, 988 continue at step 5.
    • For Model 983 continue at step 6.
  2. For Models 941, 951, inspect the I2C Y-cable for damage or misplugging. The cable can be hot-plugged. Continue at step 7.
    Table 4. Model 941, 951 I2C Y-cables
    CEC enclosure CEC enclosure connector1 Upper RPC card connector2 Lower RPC card connector2
    Upper P1-I2C C1-T210 C2-T210
    Lower P1-I2C C1-T214 C2-T214

    Note 1: See Figure 5.

    Note 2: See Figure 1.
  3. For Model 961, inspect the I2C cable for damage or misplugging. The cable can be hot-plugged. Continue at step 7.
    Table 5. Model 961 I2C cables
    CEC enclosure CEC enclosure connector1 Upper RPC card connector2 Lower RPC card connector2
    Upper P1-C1-C1-T1 C1-J210 n/a
    Upper P1-C1-C1-T2 n/a C2-J210
    Lower P1-C1-C1-T1 C1-J214  
    Lower P1-C1-C1-T2   C2-J214

    Note 1: See Figure 6.

    Note 2: See Figure 2.
  4. For Models 980, 981, 984, 985, and 986, inspect the I2C cable for damage or misplugging. The cable can be hot-plugged. Continue at step 7.
    Table 6. Model 980, 981, 983, 984, 985, 986 I2C cables
    CEC enclosure CEC enclosure connector1 RPC card connector2
    Upper P1-C8-T1 C1-J210
    Upper P1-C8-T2 C2-J210
    Lower P1-C8-T1 C1-J214
    Lower P1-C8-T2 C2-J214

    Note 1: See Figure 7.

    Note 2: See Figure 3, Figure 4.
  5. For Models 982 and 988, inspect the I2C cable for damage or misplugging. The cable can be hot-plugged. Continue at step 7.
    Table 7. Model 982, 988 I2C cables
    CEC enclosure CEC enclosure connector1 RPC card connector2
    Upper P1-C6-T1 C1-J210
    Upper P1-C6-T2 C2-J210
    Lower P1-C6-T1 C1-J214
    Lower P1-C6-T2 C2-J214

    Note 1: See Figure 8.

    Note 2: See Figure 3.
  6. For Model 983, inspect the I2C cable for damage or misplugging. The cable can be hot-plugged. Continue at step 7.
    Table 8. Model 983 I2C cables
    CEC enclosure CEC enclosure connector1 RPC card connector2
    Upper P1-C8-T1 C1-J204
    Upper P1-C8-T2 C2-J204
    Lower P1-C8-T1 C1-J206
    Lower P1-C8-T2 C2-J206

    Note 1: See Figure 7.

    Note 2: See Figure 4.
  7. Determine which RPC is fenced.
    • If an RPC card is listed in the serviceable event FRU list, then it is fenced.
    • If an RPC card is not listed in the serviceable event FRU list, use MAP1100 View storage facility state (end of call) to display fenced resources and then click Details to determine which RPC is fenced.
    • If no RPC shows as fenced, check for codes XE or XF on the status display of both DC-UPSs. Use Table 9 to select which RPC card to use for the next step.
      Note: Any combination of codes may be displayed. You are only concerned with XE and XF.
      Table 9. RPC card to pseudo-repair when none are fenced
      DC-UPS status display RPC card to pseudo-repair in next step
      Both DC-UPSs show XF RPC2 card (R1-C2)
      Both DC-UPSs show XE RPC1 card (R1-C1)
      Any other condition, including:
      XE on one DC-UPS (but not both)
      XF on one DC-UPS (but not both)      		 RPC1 card (R1-C1)
  8. Do a pseudo repair of the fenced RPC card. This resets the existing RPC card without replacing it.
  9. Was the FRU verification successful?
    • Yes, go to step 10.
    • No, go to step 11.
  10. Seating the cable corrected the problem. Exit this MAP and close related serviceable events.
  11. Determine whether the control panel is causing the problem. Use the ASM interface to logically and physically remove the CEC enclosure control panel. Refer to MAP4110 Exchange the CEC enclosure control panel (concurrent).
    Important: The CEC enclosure must not be power cycled or rebooted while the control panel is removed, to ensure that the load of the functional code succeeds.
  12. Do a pseudo repair of the fenced RPC card. This resets the existing RPC card without replacing it.
  13. Was the FRU verification successful?
    • Yes, go to step 14.
    • No, go to step 15.
  14. The control panel is causing the failure when it is installed. Replace the control panel.
  15. The CEC enclosure control panel is not failing. Use MAP4110 Exchange the CEC enclosure control panel (concurrent) to reinstall the existing control panel.
  16. Replace the remaining FRUs in the serviceable event FRU list. If there are none, then the possible failing FRUs are listed in alphabetical order below. (Refer to MAP1230 Replace a FRU without using a serviceable event.)
    • Models 941, 951:
      • CEC enclosure service processor
      • CEC enclosure I/O backplane assembly
      • RPC card to CEC enclosure cable (I2C Y-Cable)
      • RPC1 card (only when it is fenced; use MAP1100 to display fenced resources)
      • RPC2 card (only when it is fenced; use MAP1100 to display fenced resources)
    • Model 961:
      • CEC enclosure system backplane (Model 961) (contains service processor function)
      • CEC enclosure I2C card
      • RPC card to CEC enclosure I2C cable
      • RPC1 card (only when it is fenced; use MAP1100 to display fenced resources)
      • RPC2 card (only when it is fenced; use MAP1100 to display fenced resources)
    • Models 980, 981, 983, 984, 985, 986:
      • CEC enclosure system backplane (Model 98x) (contains service processor function)
      • CEC enclosure I2C card
      • RPC card to CEC enclosure I2C cable
      • RPC1 card (only when it is fenced; use MAP1100 to display fenced resources)
      • RPC2 card (only when it is fenced; use MAP1100 to display fenced resources)
    • Models 982, 988:
      • CEC enclosure service processor card
      • CEC enclosure I2C card
      • RPC card to CEC enclosure I2C cable
      • RPC1 card (only when it is fenced; use MAP1100 to display fenced resources)
      • RPC2 card (only when it is fenced; use MAP1100 to display fenced resources)

MAP2220 Section-6 (One CEC communication failure with one RPC card)

About this task

One CEC enclosure has a communication failure through the service processor to one RPC card. The RPC card is not reporting any communication failures to the service processor.

Procedure

  1. Determine whether an LPAR is fenced (not all resources running) using MAP1100 Section-16, LPARs IMLed (CPSS).
    Note: One of the two RPC cards should already be fenced.

    Is an LPAR fenced?

  2. Use the SRC in the serviceable event that sent you here to determine the failing communication interface. For Models 941, 951, use Table 10. For Model 961, use Table 11. For Model 98x, use Table 12.
    • Models 941, 951:
      1. Inspect the RPC card to CEC enclosure cable (I2C Y-Cable) for damage or misplugging. Ensure it is fully seated by pressing on each connector.
      2. If a problem is found, the cable can be hot-plugged.
      3. Most likely one of the RPC cards is fenced. Before this service action is complete, if that RPC card is not replaced for another reason, then that RPC card needs to go through a pseudo repair. A pseudo repair means the parts exchange process for that RPC card is used to create a reset to that card without physically replacing that RPC card. MAP1100 Section-27, Fenced Resources can be used to display fenced resources.
      4. Continue at the next step.
    Table 10. SRC communication interface-failure resources (Models 941, 951)
    SRC Communication failure resources Y-Cable Connections for this SRC
    BE190014 CEC enclosure 1 (upper), failure to communicate with RPC1 card (right viewed from rear)
    RPC card to CEC enclosure cable
    RPC card to PPSs Y-cable
    BE190016 CEC enclosure 1 (upper), failure to communicate with RPC2 card (left viewed from rear)
    BE190015 CEC enclosure 2 (lower), failure to communicate with RPC1 card (right viewed from rear)
    RPC card to CEC enclosure cable
    RPC card to PPSs Y-cable
    BE190017 CEC enclosure 2 (lower), failure to communicate with RPC2 card (left viewed from rear)
    • Model 961:
      1. Inspect the RPC card to CEC enclosure cable for damage or misplugging. Ensure it is fully seated by pressing on each connector.
      2. If a problem is found, the cable can be hot-plugged.
      3. Most likely one of the RPC cards is fenced. Before this service action is complete, if that RPC card is not replaced for another reason, then that RPC card needs to go through a pseudo repair. A pseudo repair means the parts exchange process for that RPC card is used to create a reset to that card without physically replacing that RPC card. MAP1100 Section-27, Fenced Resources can be used to display fenced resources.
      4. Continue at the next step.
    Table 11. SRC communication interface-failure resources (Model 961)
    SRC Communication failure resources Y-Cable Connections for this SRC
    BE190014 CEC enclosure 1 (upper), failure to communicate with RPC1 card (upper)
    CEC enclosure to RPC cards Y-cable
    RPC card to DC-UPS Y-cable
    BE190016 CEC enclosure 1 (upper), failure to communicate with RPC2 card (lower)
    CEC enclosure to RPC cards Y-cable
    RPC card to DC-UPS Y-cable
    BE190015 CEC enclosure 2 (lower), failure to communicate with RPC1 card (upper)
    CEC enclosure to RPC cards Y-cable
    RPC card to DC-UPS Y-cable
    BE190017 CEC enclosure 2 (lower), failure to communicate with RPC2 card (lower)
    CEC enclosure to RPC cards Y-cable
    RPC card to DC-UPS Y-cable
    • Model 98x:
      1. Inspect the RPC card to CEC enclosure cable for damage or misplugging. Ensure it is fully seated by pressing on each connector.
      2. If a problem is found, the cable can be hot-plugged.
      3. Most likely one of the RPC cards is fenced. Before this service action is complete, if that RPC card is not replaced for another reason, then that RPC card needs to go through a pseudo repair. A pseudo repair means the parts exchange process for that RPC card is used to create a reset to that card without physically replacing that RPC card. MAP1100 Section-27, Fenced Resources can be used to display fenced resources.
      4. Continue at the next step.
    Table 12. SRC communication interface-failure resources (Model 98x)
    SRC Communication failure resources Y-Cable Connections for this SRC
    BE190014 CEC enclosure 1 (upper), failure to communicate with RPC1 card (left)
    CEC enclosure to RPC cards Y-cable
    • Upper CEC enclosure P1-C8-T1 (Figure 7)
    • For models 982, 988, upper CEC enclosure P1-C6-T1 (Figure 8)
    • RPC1 card C1-J210 (Figure 3)
    RPC card to DC-UPS Y-cable
    BE190016 CEC enclosure 1 (upper), failure to communicate with RPC2 card (right)
    CEC enclosure to RPC cards Y-cable
    • Upper CEC enclosure P1-C8-T2 (Figure 7)
    • For models 982, 988, upper CEC enclosure P1-C6-T2 (Figure 8)
    • RPC2 card C2-J210 (Figure 3)
    RPC card to DC-UPS Y-cable
    BE190015 CEC enclosure 2 (lower), failure to communicate with RPC1 card (left)
    CEC enclosure to RPC cards Y-cable
    • Lower CEC enclosure P1-C8-T1 (Figure 7)
    • For models 982, 988, lower CEC enclosure P1-C6-T1 (Figure 8)
    • RPC1 card C1-J214 (Figure 3)
    RPC card to DC-UPS Y-cable
    BE190017 CEC enclosure 2 (lower), failure to communicate with RPC2 card (right)
    CEC enclosure to RPC cards Y-cable
    • Lower CEC enclosure P1-C8-T2 (Figure 7)
    • For models 982, 988, lower CEC enclosure P1-C6-T2 (Figure 8)
    • RPC2 card C2-J214 (Figure 3)
    RPC card to DC-UPS Y-cable
  3. Display serviceable events that need repair.

    Is there any other serviceable event with one or more of the same FRUs as listed in the serviceable event that sent you to this MAP?

    • Yes, exit this MAP and repair the other serviceable event.
      • If the repair of the other serviceable event is successful, it closes automatically. Manually close the serviceable event that sent you here.
      • If the repair is not successful, go to step 4.
    • No, go to step 4.
  4. Replace hardware FRUs listed in the serviceable event that sent you here until the problem is repaired.
    If no hardware FRUs are listed, the possible failing FRUs are listed here. To exchange a FRU that is not listed in a serviceable event, refer to MAP1230 Replace a FRU without using a serviceable event.
    • Models 941, 951:
      • CEC enclosure service processor
      • CEC enclosure I/O backplane assembly
      • CEC enclosure to RPC card (I2C Y-Cable)
      • RPC1 card (only when it is fenced; use MAP1100 to display fenced resources)
      • RPC2 card (only when it is fenced; use MAP1100 to display fenced resources)
      • RPC card to primary power supply cable (to both PPS in Rack-1)
    • Model 961:
      • CEC enclosure system backplane (Model 961) (contains service processor function)
      • CEC enclosure I2C card
      • RPC card to CEC enclosure I2C cable
      • RPC1 card (only when it is fenced; use MAP1100 to display fenced resources)
      • RPC2 card (only when it is fenced; use MAP1100 to display fenced resources)
      • RPC card to DC-UPS cable (in Rack-1 only)
    • Models 980, 981, 984, 985, 986:
      • CEC enclosure system backplane (contains service processor function)
      • CEC enclosure I2C card
      • RPC card to CEC enclosure I2C cable
      • RPC1 card (only when it is fenced; use MAP1100 to display fenced resources)
      • RPC2 card (only when it is fenced; use MAP1100 to display fenced resources)
      • RPC card to DC-UPS cable (in Rack-1 only)
    • Model 983:
      • CEC enclosure system backplane (contains service processor function)
      • CEC enclosure I2C card
      • RPC card to CEC enclosure I2C cable
      • RPC1 card (only when it is fenced; use MAP1100 to display fenced resources)
      • RPC2 card (only when it is fenced; use MAP1100 to display fenced resources)
    • Models 982, 988:
      • CEC enclosure service processor card
      • CEC enclosure I2C card
      • RPC card to CEC enclosure I2C cable
      • RPC1 card (only when it is fenced; use MAP1100 to display fenced resources)
      • RPC2 card (only when it is fenced; use MAP1100 to display fenced resources)
      • RPC card to DC-UPS cable (in Rack-1 only)