Failure to upload or download data in objectstore

Description

This issue may occur when the object-store state is in a degraded state due to a placement group (PG) inconsistency.

Verify if the problem is indeed related to rook-ceph PG inconsistency by running the following commands:

export KUBECONFIG=/etc/rancher/rke2/rke2.yaml PATH=$PATH:/var/lib/rancher/rke2/bin
ROOK_CEPH_TOOLS=$(kubectl -n rook-ceph get pods | grep rook-ceph-tools)
kubectl -n rook-ceph exec -it $ROOK_CEPH_TOOLS -- ceph statusexport KUBECONFIG=/etc/rancher/rke2/rke2.yaml PATH=$PATH:/var/lib/rancher/rke2/bin
ROOK_CEPH_TOOLS=$(kubectl -n rook-ceph get pods | grep rook-ceph-tools)
kubectl -n rook-ceph exec -it $ROOK_CEPH_TOOLS -- ceph status

If the problem is related to a rook-ceph PG inconsistency, the output should contain the following messages:

....
....
Possible data damage: X pgs inconsistent
....
....
X active+clean+inconsistent
....
........
....
Possible data damage: X pgs inconsistent
....
....
X active+clean+inconsistent
....
....

Solution

To repair the inconsistent PG, take the following steps:

Exec to rook-ceph tools:

kubectl -n rook-ceph exec -it $ROOK_CEPH_TOOLS -- shkubectl -n rook-ceph exec -it $ROOK_CEPH_TOOLS -- sh

Trigger the rook-ceph garbage collector process. Wait until the process is complete.
```
radosgw-admin gc processradosgw-admin gc process
```

Find a list of active+clean+inconsistent PGs:

ceph health detail
# output of this command be like
# ....
# pg <pg-id> is active+clean+inconsistent, acting ..
# pg <pg-id> is active+clean+inconsistent, acting ..
# ....
#ceph health detail
# output of this command be like
# ....
# pg <pg-id> is active+clean+inconsistent, acting ..
# pg <pg-id> is active+clean+inconsistent, acting ..
# ....
#

Trigger a deep scrub on the PGs one at a time. This command takes few minutes to run, depending on the PG size.
```
ceph pg deep-scrub <pg-id>ceph pg deep-scrub <pg-id>
```

Watch the scrubbing status:

ceph -w | grep <pg-id>ceph -w | grep <pg-id>

Check the PG scrub status. If the PG scrub is successful, the PG status should be active+clean+inconsistent.
```
ceph health detail | grep <pg-id>ceph health detail | grep <pg-id>
```

Repair the PG:

ceph pg repair <pg-id>ceph pg repair <pg-id>

Check the PG repair status. The PG ID should be removed from the active+clean+inconsistent list if the PG is repaired successfully.
```
ceph health detail | grep <pg-id>ceph health detail | grep <pg-id>
```
Repeat steps 3 to 8 for the rest of the inconsistent PG.

On this page

Description
Solution

Was this page helpful?

PREVIOUSStorage troubleshooting

NEXTPVC resize does not heal Ceph

Support and Services

Get The Help You Need

UiPath Academy

Learning RPA - Automation Courses

UiPath Forum

UiPath Community Forum

Trust and Security

Cookies Policy