Introduction

Due to an openstack-cli limitation removing S3 buckets with more then 10 000 objects will fail when using the command:

openstack container delete --recursive <<bucket_name>>

The command will fail with an error:

Conflict (HTTP 409) (Request-ID: tx00000000000001bb5e8e5-006135c488-35bc5d520-dias_default) clean_up DeleteContainer: Conflict (HTTP 409) (Request-ID:

Conflict means "tried to remove not empty bucket"

Recommended approach

It's recommended to use swift-cli because it's:

  - not being limited to deleting only 10000 objects

  - have multiple threads deleting objects

If you do not have python3-swiftclient installed please type this command first:

sudo apt install python3-swiftclient

And then use swift-cli 

swift delete <<bucket>>

Special case

In really rare situations if the bucket is still not deleted, you should send to support the logs collected with the command:

openstack -vv container delete --recursive 2&>1 | tee container_delete.log

In the logs we are looking for GET of object list and DELETE requests.

In normal circumstances openstack-cli should:

  1. ask for info about a bucket GET /swift/v1/AUTH_<<auth_id>>/<<bucket-name>>?format=json

  2. get response with list  of objects in a bucket, for example:

# Ask for info about bucket REQ: curl -g -i -X GET "https://s3.waw2-1.cloudferro.com/swift/v1/AUTH_e31d0ad8bb8b45968ac5d0167b0b32a7/SPN_2018_L2A-S2?format=json" -H "User-Agent: openstacksdk/0.56.0 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.8.10" -H "X-Auth-Token: {SHA256}8043d0760e0eba8c565d775d73be2a11c2a6614b8d249097a0b6ae14f348217a" Starting new HTTPS connection (1): s3.waw2-1.cloudferro.com:443 https://s3.waw2-1.cloudferro.com:443 "GET /swift/v1/AUTH_e31d0ad8bb8b45968ac5d0167b0b32a7/SPN_2018_L2A-S2?format=json HTTP/1.1" 200 2970919 RESP: [200] accept-ranges: bytes content-length: 2970919 content-type: application/json; charset=utf-8 date: Mon, 06 Sep 2021 07:19:01 GMT x-container-bytes-used: 9954559438802 x-container-bytes-used-actual: 9955001249792 x-container-object-count: 340750 x-openstack-request-id: tx00000000000000b088847-006135c0e4-36df76306-dias_default x-storage-policy: default-placement x-timestamp: 1556580242.32837 x-trans-id: tx00000000000000b088847-006135c0e4-36df76306-dias_default # List of items in a bucket RESP BODY: [{"name":"S2A_MSIL2A_20180706T110621_N0206_R137_T29TQH_20180706T132104.SAFE/S2A_OPER_SSC_L2VALD_29TQH____20180706.DBL.DIR/PRIVATE/S2A_OPER_SSC_LUTANX_L2VALD_29TQH____20180706_LTC.DBL.DIR/S2A_OPER_SSC_LUTANX_L2VALD_29TQH____20171108_LTC.DBL.mha","hash":"7f385b84120fa1758215d93e79957d29","bytes":16333,"last_modified":"2019-05-03T22:26:50.632Z"},{"name":"S2A_MSIL2A_20180706T110621_N0206_R137_T29TQH_20180706T132104.SAFE/S2A_OPER_SSC_L2VALD_29TQH____20180706.DBL.DIR/PRIVATE/S2A_OPER_SSC_LUTANX_L2VALD_29TQH____20180706_LTC.DBL.DIR/S2A_OPER_SSC_LUTANX_L2VALD_29TQH____20171118_LTC.DBL.mha","hash":"10067a14556289ce7bbaf99bfafb7028","bytes":16333,"last_modified":"2019-05-03T22:26:51.151Z"},{"name":"S2A_MSIL2A_20180706T110621_N0206_R137_T29TQH_20180706T132104.SAFE/S2A_OPER_SSC_L2VALD_29TQH____20180706.DBL.DIR/PRIVATE/S2A_OPER_SSC_LUTANX_L2VALD_29TQH____20180706_LTC.DBL.DIR/

  3. delete every item in bucket

  4. delete bucket

Here is the example log of special case:

Starting new HTTPS connection (1): s3.waw2-1.cloudferro.com:443 # GET info about Bucket https://s3.waw2-1.cloudferro.com:443 "GET /swift/v1/AUTH_e31d0ad8bb8b45968ac5d0167b0b32a7/COG2019?format=json HTTP/1.1" 200 2 RESP: [200] accept-ranges: bytes content-length: 2 content-type: application/json; charset=utf-8 date: Tue, 14 Sep 2021 05:38:12 GMT x-container-bytes-used: 737440062 x-container-bytes-used-actual: 737443840 x-container-object-count: 141 x-container-read: .r:*,.rlistings x-openstack-request-id: tx00000000000002138956e-0061403533-35bc5d520-dias_default x-storage-policy: default-placement x-timestamp: 1560926357.15375 x-trans-id: tx00000000000002138956e-0061403533-35bc5d520-dias_default RESP BODY: [] GET call to https://s3.waw2-1.cloudferro.com/swift/v1/AUTH_e31d0ad8bb8b45968ac5d0167b0b32a7/COG2019?format=json used request id tx00000000000002138956e-0061403533-35bc5d520-dias_default # DELETE CALL REQ: curl -g -i -X DELETE https://s3.waw2-1.cloudferro.com/swift/v1/AUTH_e31d0ad8bb8b45968ac5d0167b0b32a7/COG2019 -H "User-Agent: openstacksdk/0.56.0 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.8.10" -H "X-Auth-Token: {SHA256}9a49a9b023d3736df9f50603c36e925912fcb429a6697d55a57472110623a87f"

In the above log you can see that:

  - a container is not empty, it still has objects as stated by HTTP header.

x-container-object-count: 141

the list of objects in the bucket is empty (in contrast to normal circumstances)

RESP BODY: []

In this  situation openstack-cli didn't get a list of objects even though bucket is not empty, and openstack-cli is trying to delete the whole bucket right away and omitting the step where openstack-cli should delete objects residing in a bucket.

This is mostly because of a damaged index of a bucket. CREODIAS Support Team can help with such situation as resolving this problem requires direct command execution in Ceph (software-defined storage).