Uploaded image for project: 'eZ Publish / Platform'
  1. eZ Publish / Platform
  2. EZP-25792

Index is not properly updated when Location is deleted

    Details

      Description

      Deleting a location deletes a whole subtree. In Solr search engine we only update the deleted Location and it's Content, while the whole affected subtree should be properly updated.

        Activity

        Petar Spanja (Inactive) created issue -
        Show
        Kamil Ronewicz (Inactive) added a comment - PR: https://github.com/ezsystems/ezpublish-kernel/pull/1658 and https://github.com/ezsystems/ezplatform-solr-search-engine/pull/46
        Kamil Ronewicz (Inactive) made changes -
        Field Original Value New Value
        Status Open [ 1 ] Confirmed [ 10037 ]
        Kamil Ronewicz (Inactive) made changes -
        Status Confirmed [ 10037 ] Open [ 1 ]
        André Rømcke made changes -
        Status Open [ 1 ] Confirmed [ 10037 ]
        André Rømcke made changes -
        Status Confirmed [ 10037 ] Backlog [ 10000 ]
        André Rømcke made changes -
        Status Backlog [ 10000 ] Development [ 3 ]
        Assignee André Rømcke [ andre.romcke@ez.no ]
        André Rømcke made changes -
        Assignee André Rømcke [ andre.romcke@ez.no ] Kamil Ronewicz [ kamil.roniewicz@ez.no ]
        André Rømcke made changes -
        Status Development [ 3 ] Development Review [ 10006 ]
        Assignee Kamil Ronewicz [ kamil.roniewicz@ez.no ] André Rømcke [ andre.romcke@ez.no ]
        Show
        André Rømcke added a comment - Merged in: https://github.com/ezsystems/ezplatform-solr-search-engine/commit/ce42dc711090038d1ebeb2041d3a986acfa77c62 Kernel (tests): https://github.com/ezsystems/ezpublish-kernel/commit/8df550b593dbcc2eb30add3e09ae095806082e44
        André Rømcke made changes -
        Status Development Review [ 10006 ] Documentation Review done [ 10011 ]
        Fix Version/s 5.4.7 [ 14519 ]
        Fix Version/s 1.4.0 [ 14528 ]
        Fix Version/s 1.3.2 [ 14530 ]
        Assignee André Rømcke [ andre.romcke@ez.no ]
        Rui Silva (Inactive) made changes -
        Status Documentation Review done [ 10011 ] QA [ 10008 ]
        Hide
        Rui Silva (Inactive) added a comment - - edited

        Edited comment:

        [~kamil.ronewicz@ez.no], Petar Spanja, QA is trying to reproduce this issue but is not able to.
        Having solr up and running,
        and the SoftCommit option configured to 100ms instead of the default -1 value,
        if I create the following content tree structure:

        - eZ Platform (already existing)
        --- Parent
        ----- Child
        ------- Grandchild
        

        and index the contents:

        php app/console ezplatform:solr_create_index
        Indexing Content...
         16/16 [============================] 100%
        

        (13 contents already exist on default "clean" installation, hence indexing 16 contents: 13 + the 3 new ones)
        I go to the Solr admin interface onto the Core's Statistics view, where I can see how many contents are indexed.
        It shows me:
        Num Docs: 32
        Max Doc: 32
        which is double the value because it indexes both contents and locations as far as I know.

        Then, I proceed to delete (Send to Trash) only content "Parent", thus removing "Child" and "Grandchild" by aggregation.
        I check the indexed content again on my Core's Statistics view, and I still see:
        Num Docs: 32
        Max Doc: 32
        even though I had to manually click "Reload" on Solr admin interface for some reason..
        Even though, after reloading it, it still shows the same amount as before as if it did not update anything at all.

        Show
        Rui Silva (Inactive) added a comment - - edited Edited comment: [~kamil.ronewicz@ez.no] , Petar Spanja , QA is trying to reproduce this issue but is not able to. Having solr up and running, and the SoftCommit option configured to 100ms instead of the default -1 value, if I create the following content tree structure: - eZ Platform (already existing) --- Parent ----- Child ------- Grandchild and index the contents: php app/console ezplatform:solr_create_index Indexing Content... 16/16 [============================] 100% (13 contents already exist on default "clean" installation, hence indexing 16 contents: 13 + the 3 new ones) I go to the Solr admin interface onto the Core's Statistics view, where I can see how many contents are indexed. It shows me: Num Docs: 32 Max Doc: 32 which is double the value because it indexes both contents and locations as far as I know. Then, I proceed to delete (Send to Trash) only content "Parent", thus removing "Child" and "Grandchild" by aggregation. I check the indexed content again on my Core's Statistics view, and I still see: Num Docs: 32 Max Doc: 32 even though I had to manually click "Reload" on Solr admin interface for some reason.. Even though, after reloading it, it still shows the same amount as before as if it did not update anything at all.
        Hide
        Paulo Nunes (Inactive) added a comment -

        On eZ Publish 5.4.6, I'm having the same results as Rui,
        I tried on two situations: having ezplatformsearch extension made by NetGen, as referred in solr documentation

        With netgen extension

        I don't even need the patch from the current issue. Solr always indexes the removed content, including the removed sub-items

        Without Netgen extension

        With or without applying the patch from the current issue, I'm having the same result as reported by Rui.

        Show
        Paulo Nunes (Inactive) added a comment - On eZ Publish 5.4.6, I'm having the same results as Rui, I tried on two situations: having ezplatformsearch extension made by NetGen , as referred in solr documentation With netgen extension I don't even need the patch from the current issue. Solr always indexes the removed content, including the removed sub-items Without Netgen extension With or without applying the patch from the current issue, I'm having the same result as reported by Rui .
        Hide
        Rui Silva (Inactive) added a comment -

        Since this is apparently not fixed, I'm sending back to dev needed.

        Show
        Rui Silva (Inactive) added a comment - Since this is apparently not fixed, I'm sending back to dev needed.
        Rui Silva (Inactive) made changes -
        Status QA [ 10008 ] InputQ [ 10001 ]
        Assignee Rui Silva [ rui.silva@ez.no ]
        Petar Spanja (Inactive) made changes -
        Assignee Petar Spanja [ petar.spanja@ez.no ]
        Hide
        Petar Spanja (Inactive) added a comment - - edited

        [~rui.silva@ez.no], Paulo Nunes

        Let me know how are you testing this, through legacy admin UI? (Since it is tagged with 5.4.7 as fix version)

        If that is the case – we do not have a mechanism to connect legacy admin UI with indexing in new stack.
        ezplatformsearch extension is external and I don't think it should be tested here, so this should be tested through API only.

        Show
        Petar Spanja (Inactive) added a comment - - edited [~rui.silva@ez.no] , Paulo Nunes Let me know how are you testing this, through legacy admin UI? (Since it is tagged with 5.4.7 as fix version) If that is the case – we do not have a mechanism to connect legacy admin UI with indexing in new stack. ezplatformsearch extension is external and I don't think it should be tested here, so this should be tested through API only.
        Petar Spanja (Inactive) made changes -
        Status InputQ [ 10001 ] Development [ 3 ]
        Hide
        Petar Spanja (Inactive) added a comment -

        QA: ensure this is tested in a correct way (see my previous comment).

        Show
        Petar Spanja (Inactive) added a comment - QA: ensure this is tested in a correct way (see my previous comment).
        Petar Spanja (Inactive) made changes -
        Status Development [ 3 ] Development Review done [ 10028 ]
        Assignee Petar Spanja [ petar.spanja@ez.no ]
        Petar Spanja (Inactive) made changes -
        Status Development Review done [ 10028 ] Documentation Review done [ 10011 ]
        Rui Silva (Inactive) made changes -
        Status Documentation Review done [ 10011 ] QA [ 10008 ]
        Hide
        Rui Silva (Inactive) added a comment -

        Petar Spanja, QA has tested this on a 5.4 and on master. On a 5.4 we were able to reproduce the issue and verify the fix solves it.
        However on a master, the fix is not working.
        I hereby transcribe the testcase / steps, plus expected result, that I am using to reproduce this.
        In attachment you can find the bundle I used the public API from to test this.

        1. Install Solr and set it to run, with softAutoCommit set to 100:
        vi solr/solr/collection1/conf/solrconfig.xml
        search for softAutoCommit and change the value;

        2. Restart Solr;

        3. Generate a new bundle:
        php ezpublish/console generate:bundle --no-interaction --format=yml --dir=src --namespace=EzSystems/Ezp25792Bundle
        php app/console generate:bundle --no-interaction --format=yml --dir=src --namespace=EzSystems/Ezp25792Bundle

        4. Import into / replace the contents of the generated bundle with the ones from the attached bundle, to make the commands needed for the test available;

        5. Execute:
        php ezpublish/console testsSystem:ezp25792:createContent 2 Parent
        php app/console testsSystem:ezp25792:createContent 2 Parent
        where 2 is the location id under which the content is supposed to be created, and "Parent" is its name;
        You should get something like:
        Content 'Parent' content id: 135
        Content 'Parent' location id: 131

        6. Execute now:
        php ezpublish/console testsSystem:ezp25792:createContent 131 Child
        php app/console testsSystem:ezp25792:createContent 131 Child
        i.e., create "Child" under content created previously by now passing to the command as parameter the location id returned from previous command.
        Expect to get something like:
        Content 'Parent' content id: 136
        Content 'Parent' location id: 132

        7. Execute yet again:
        php ezpublish/console testsSystem:ezp25792:createContent 132 Grandchild
        php app/console testsSystem:ezp25792:createContent 132 Grandchild
        Expect to get something like:
        Content 'Parent' content id: 137
        Content 'Parent' location id: 133

        Now you have the following content structure:
        <Home>
        -> Parent
        ---> Child
        -----> Grandchild

        8. Now index the contents with solr:
        php ezpublish/console ezplatform:solr_create_index
        php app/console ezplatform:solr_create_index

        9. Verify on solr admin that on Statistics the Num Docs and Max Docs values you have reflect the indexed content values (number = contents + locations, so, two per content object);

        10. Now execute:
        php ezpublish/console testsSystem:ezp25792:ezp25792removeSubtree 131
        php app/console testsSystem:ezp25792:ezp25792removeSubtree 131
        i.e., remove the "Parent" content given its location

        11. Now recheck the Solr admin at Statistics again, the same values, and verify that both values decreased by 6, (content object + location per each of the three contents affected to "Parent"'s content subtree).

        Show
        Rui Silva (Inactive) added a comment - Petar Spanja , QA has tested this on a 5.4 and on master. On a 5.4 we were able to reproduce the issue and verify the fix solves it. However on a master, the fix is not working. I hereby transcribe the testcase / steps, plus expected result, that I am using to reproduce this. In attachment you can find the bundle I used the public API from to test this. 1. Install Solr and set it to run, with softAutoCommit set to 100: vi solr/solr/collection1/conf/solrconfig.xml search for softAutoCommit and change the value; 2. Restart Solr; 3. Generate a new bundle: php ezpublish/console generate:bundle --no-interaction --format=yml --dir=src --namespace=EzSystems/Ezp25792Bundle php app/console generate:bundle --no-interaction --format=yml --dir=src --namespace=EzSystems/Ezp25792Bundle 4. Import into / replace the contents of the generated bundle with the ones from the attached bundle, to make the commands needed for the test available; 5. Execute: php ezpublish/console testsSystem:ezp25792:createContent 2 Parent php app/console testsSystem:ezp25792:createContent 2 Parent where 2 is the location id under which the content is supposed to be created, and "Parent" is its name; You should get something like: Content 'Parent' content id: 135 Content 'Parent' location id: 131 6. Execute now: php ezpublish/console testsSystem:ezp25792:createContent 131 Child php app/console testsSystem:ezp25792:createContent 131 Child i.e., create "Child" under content created previously by now passing to the command as parameter the location id returned from previous command. Expect to get something like: Content 'Parent' content id: 136 Content 'Parent' location id: 132 7. Execute yet again: php ezpublish/console testsSystem:ezp25792:createContent 132 Grandchild php app/console testsSystem:ezp25792:createContent 132 Grandchild Expect to get something like: Content 'Parent' content id: 137 Content 'Parent' location id: 133 Now you have the following content structure: <Home> -> Parent ---> Child -----> Grandchild 8. Now index the contents with solr: php ezpublish/console ezplatform:solr_create_index php app/console ezplatform:solr_create_index 9. Verify on solr admin that on Statistics the Num Docs and Max Docs values you have reflect the indexed content values (number = contents + locations, so, two per content object); 10. Now execute: php ezpublish/console testsSystem:ezp25792:ezp25792removeSubtree 131 php app/console testsSystem:ezp25792:ezp25792removeSubtree 131 i.e., remove the "Parent" content given its location 11. Now recheck the Solr admin at Statistics again, the same values, and verify that both values decreased by 6, (content object + location per each of the three contents affected to "Parent"'s content subtree).
        Rui Silva (Inactive) made changes -
        Attachment Ezp25792Bundle.tar.gz [ 26892 ]
        Rui Silva (Inactive) made changes -
        Flagged Impediment [ 10000 ]
        Hide
        Petar Spanja (Inactive) added a comment -

        [~rui.silva@ez.no]

        When testing Solr, you can use commit(true) method on the Solr search handler to ensure that data is written to stable Solr storage.
        You can get this service from container, through ID `ezpublish.spi.search`.

        Show
        Petar Spanja (Inactive) added a comment - [~rui.silva@ez.no] When testing Solr, you can use commit(true) method on the Solr search handler to ensure that data is written to stable Solr storage. You can get this service from container, through ID `ezpublish.spi.search`.
        Hide
        Rui Silva (Inactive) added a comment - - edited

        Petar Spanja, I updated the jira with a newer version of my bundle replacing the former one, which is now using the commit( true ).
        It still does not work, I don't know if it doesn't somehow conflict now with the solrconfig.xml options autoSoftCommit and autoCommit.
        Whether the later is set to "true" or "false", nothing changes, using the commit() after the removal of the subtree root content, it still does not work. No cache-related problem and did it several times.

        Show
        Rui Silva (Inactive) added a comment - - edited Petar Spanja , I updated the jira with a newer version of my bundle replacing the former one, which is now using the commit( true ). It still does not work, I don't know if it doesn't somehow conflict now with the solrconfig.xml options autoSoftCommit and autoCommit. Whether the later is set to "true" or "false", nothing changes, using the commit() after the removal of the subtree root content, it still does not work. No cache-related problem and did it several times.
        Rui Silva (Inactive) made changes -
        Attachment Ezp25792Bundle.tar.gz [ 26892 ]
        Rui Silva (Inactive) made changes -
        Attachment Ezp25792Bundle.tar.gz [ 26893 ]
        Hide
        Petar Spanja (Inactive) added a comment -

        [~rui.silva@ez.no] Why are you "muting" NotFoundException on $locationService->deleteLocation( $location );?

        The cause for stuff not being deleted from index is that cache purge fails with NotFoundException.
        This happens because the Location and Content do not exist at the point where cache purger tries to load ContentInfo and Locations so that it can clear the correct cache. This is the place:

        https://github.com/ezsystems/ezpublish-kernel/blob/master/eZ/Publish/Core/MVC/Symfony/Cache/Http/InstantCachePurger.php#L68

        I knew about this issue, but somehow had an idea that it was fixed long time ago.
        Apparently it was not, TBH not sure how it's possible what we have this.

        Ping André Rømcke it would be good if you could shed some light on this.

        Show
        Petar Spanja (Inactive) added a comment - [~rui.silva@ez.no] Why are you "muting" NotFoundException on $locationService->deleteLocation( $location ); ? The cause for stuff not being deleted from index is that cache purge fails with NotFoundException . This happens because the Location and Content do not exist at the point where cache purger tries to load ContentInfo and Locations so that it can clear the correct cache. This is the place: https://github.com/ezsystems/ezpublish-kernel/blob/master/eZ/Publish/Core/MVC/Symfony/Cache/Http/InstantCachePurger.php#L68 I knew about this issue, but somehow had an idea that it was fixed long time ago. Apparently it was not, TBH not sure how it's possible what we have this. Ping André Rømcke it would be good if you could shed some light on this.
        Hide
        André Rømcke added a comment -

        Cache clearing issues is covered by EZP-24621 and it's related issues.

        QA: Please close this as we have test coverage for this specific case, and rather do the overall system testing when EZP-24621 and EZP-25003 is fixed.

        Show
        André Rømcke added a comment - Cache clearing issues is covered by EZP-24621 and it's related issues. QA: Please close this as we have test coverage for this specific case, and rather do the overall system testing when EZP-24621 and EZP-25003 is fixed.
        Hide
        Rui Silva (Inactive) added a comment -

        Petar Spanja, I wrapped it in try catch because for some reason unknown to us (we've dug around this for a while trying to figure it out) it was always throwing the NotFoundException and sometimes it deleted the object anyway, and sometimes it did not (checked though admin interface), and we did not figure out why.
        Thus, unbeknownst to us how this was or was not related to the issue itself, we decided to simply mute it in order to bypass something we could not wrap our heads around that resulted in varying outcomes.
        Where you say "The cause for stuff not being deleted from index is that cache purge fails with NotFoundException", is that the explanation then as to why the contents are not being removed? So this issue is known to only happen on a master? On a 5.4 it did not happen as QA tested it, neither (as far as I can remember) the NotFoundException.

        André Rømcke, I will QA this then for now, and then perhaps when EZP-24621 is fixed (which seems to cover a more broad scope) the testing procedure will cover this as well.

        Show
        Rui Silva (Inactive) added a comment - Petar Spanja , I wrapped it in try catch because for some reason unknown to us (we've dug around this for a while trying to figure it out) it was always throwing the NotFoundException and sometimes it deleted the object anyway, and sometimes it did not (checked though admin interface), and we did not figure out why. Thus, unbeknownst to us how this was or was not related to the issue itself, we decided to simply mute it in order to bypass something we could not wrap our heads around that resulted in varying outcomes. Where you say "The cause for stuff not being deleted from index is that cache purge fails with NotFoundException", is that the explanation then as to why the contents are not being removed? So this issue is known to only happen on a master? On a 5.4 it did not happen as QA tested it, neither (as far as I can remember) the NotFoundException. André Rømcke , I will QA this then for now, and then perhaps when EZP-24621 is fixed (which seems to cover a more broad scope) the testing procedure will cover this as well.
        Hide
        Petar Spanja (Inactive) added a comment -

        [~rui.silva@ez.no] Yes, that's the explanation, it happens because both use SignalSlot system.
        BTW you can increase command verbosity output level and you'll get stack trace.

        Show
        Petar Spanja (Inactive) added a comment - [~rui.silva@ez.no] Yes, that's the explanation, it happens because both use SignalSlot system. BTW you can increase command verbosity output level and you'll get stack trace.
        Hide
        Rui Silva (Inactive) added a comment -

        Tested and approved by QA for 5.4.
        Not approved (yet) for master though, but closing issue anyway, since known issue prevents correct testing of this on master, and a broader testing scope will cover this when EZP-24621 is fixed (See comments before for more info).

        Show
        Rui Silva (Inactive) added a comment - Tested and approved by QA for 5.4. Not approved (yet) for master though, but closing issue anyway, since known issue prevents correct testing of this on master, and a broader testing scope will cover this when EZP-24621 is fixed (See comments before for more info).
        Hide
        Rui Silva (Inactive) added a comment -

        Thanks for the info Petar Spanja. I'll have to verify that.
        That will definitely be helpful for other future issues.

        Show
        Rui Silva (Inactive) added a comment - Thanks for the info Petar Spanja . I'll have to verify that. That will definitely be helpful for other future issues.
        Rui Silva (Inactive) made changes -
        Assignee Rui Silva [ rui.silva@ez.no ]
        Status QA [ 10008 ] Closed [ 6 ]
        Resolution Fixed [ 1 ]
        Alex Schuster made changes -
        Workflow EZ* Development Workflow [ 98883 ] EZEE Development Workflow [ 125710 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Confirmed Confirmed Open Open
        3s 1 kamil.roniewicz@ez.no 06/Jun/16 1:58 PM
        Open Open Confirmed Confirmed
        19d 23h 9m 2 André Rømcke 08/Jun/16 2:44 PM
        Confirmed Confirmed Backlog Backlog
        5s 1 André Rømcke 08/Jun/16 2:44 PM
        Backlog Backlog Development Development
        3s 1 André Rømcke 08/Jun/16 2:44 PM
        Development Development Development Review Development Review
        6h 51m 1 André Rømcke 08/Jun/16 9:35 PM
        Development Review Development Review Documentation Review done Documentation Review done
        56s 1 André Rømcke 08/Jun/16 9:36 PM
        QA QA InputQ InputQ
        5d 21h 46m 1 rui.silva@ez.no 21/Jun/16 2:22 PM
        InputQ InputQ Development Development
        6d 21h 17m 1 Petar Spanja (Inactive) 28/Jun/16 11:40 AM
        Development Development Development Review done Development Review done
        44s 1 Petar Spanja (Inactive) 28/Jun/16 11:40 AM
        Development Review done Development Review done Documentation Review done Documentation Review done
        10s 1 Petar Spanja (Inactive) 28/Jun/16 11:40 AM
        Documentation Review done Documentation Review done QA QA
        7d 19h 17m 2 rui.silva@ez.no 29/Jun/16 11:59 AM
        QA QA Closed Closed
        2d 2h 37m 1 rui.silva@ez.no 01/Jul/16 2:36 PM

          People

          • Assignee:
            Unassigned
            Reporter:
            Petar Spanja (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: