Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Medium Medium
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Customer request
    • Component/s: None
    • Labels:
      None
    • Environment:

      eZ Publish 4.6 / eZ Script Monitor 1.3.0

      Description

      Editing/modifying a class with a large number of objects (e.g. 200.000) will cause eZScriptMonitor to generate an overflow of the allocated memory.

      Steps to reproduce:

      1. Activate eZScriptMonitor extension;
      2. Edit and modify a class with a large number of objects (e.g. 200.000).

        Issue Links

          Activity

          Hide
          Joao Inacio (Inactive) added a comment - - edited

          @Team:
          Suggested PR at

          I am unsure on what the concrete number of objects/versions should be, but in any case the principle should apply (batched processing)

          Show
          Joao Inacio (Inactive) added a comment - - edited @Team: Suggested PR at ezpublish-legacy: https://github.com/ezsystems/ezpublish-legacy/pull/582 ezscriptmonitor: https://github.com/ezsystems/ezscriptmonitor/pull/13 I am unsure on what the concrete number of objects/versions should be, but in any case the principle should apply (batched processing)
          Show
          Yannick Roger (Inactive) added a comment - Merged in master : ezpublish-legacy: https://github.com/ezsystems/ezpublish-legacy/commit/852775e2397da0eb0d8989b0e42d1d836e6be616 ezscriptmonitor: https://github.com/ezsystems/ezscriptmonitor/commit/eb1cd6a5cef6d5393d653981cb034fcb189cc3b9
          Hide
          Pedro Resende (Inactive) added a comment -

          While trying to reproduce this issue I've created 200 000 articles and added a new text field, ran the script monitor cronjob. After a few minutes in the Script monitor tab I've verified the progress finished, however the new field didn't appear in the articles.
          After looking a bit further I found the following error in error.log

          [ Apr 19 2013 10:54:27 ] [pedro.cleverti.qa.ezpublish.no] eZMySQLiDB:
          Query error (1206): The total number of locks exceeds the lock table size. Query: INSERT INTO ezcontentobject_attribute( contentobject_id, version, contentclassattribute_id, data_type_string, language_code, language_id )
                      SELECT a.contentobject_id, a.version, 183, 'ezisbn', a.language_code, MAX(a.language_id)
                      FROM ezcontentobject_attribute a, ezcontentobject o
                      WHERE o.id = a.contentobject_id AND
                            o.contentclass_id=2
                      GROUP BY contentobject_id,
                               version,
                               language_code
          [ Apr 19 2013 10:55:03 ] [pedro.cleverti.qa.ezpublish.no] eZMySQLiDB:
          Query error (1206): The total number of locks exceeds the lock table size. Query: UPDATE ezcontentobject_attribute,
                                           ( SELECT contentobject_id, language_code, version, contentclassattribute_id, MIN( id ) AS minid
                                           FROM ezcontentobject_attribute WHERE contentclassattribute_id = 183
                                        GROUP BY contentobject_id, language_code, version, contentclassattribute_id ) t
                                        SET ezcontentobject_attribute.id = t.minid
                                        WHERE ezcontentobject_attribute.contentobject_id = t.contentobject_id
                                        AND ezcontentobject_attribute.language_code = t.language_code
                                        AND ezcontentobject_attribute.contentclassattribute_id = 183

          I've opened a new issue -> https://jira.ez.no/browse/EZP-20737

          Show
          Pedro Resende (Inactive) added a comment - While trying to reproduce this issue I've created 200 000 articles and added a new text field, ran the script monitor cronjob. After a few minutes in the Script monitor tab I've verified the progress finished, however the new field didn't appear in the articles. After looking a bit further I found the following error in error.log [ Apr 19 2013 10:54:27 ] [pedro.cleverti.qa.ezpublish.no] eZMySQLiDB: Query error (1206): The total number of locks exceeds the lock table size. Query: INSERT INTO ezcontentobject_attribute( contentobject_id, version, contentclassattribute_id, data_type_string, language_code, language_id ) SELECT a.contentobject_id, a.version, 183, 'ezisbn', a.language_code, MAX(a.language_id) FROM ezcontentobject_attribute a, ezcontentobject o WHERE o.id = a.contentobject_id AND o.contentclass_id=2 GROUP BY contentobject_id, version, language_code [ Apr 19 2013 10:55:03 ] [pedro.cleverti.qa.ezpublish.no] eZMySQLiDB: Query error (1206): The total number of locks exceeds the lock table size. Query: UPDATE ezcontentobject_attribute, ( SELECT contentobject_id, language_code, version, contentclassattribute_id, MIN( id ) AS minid FROM ezcontentobject_attribute WHERE contentclassattribute_id = 183 GROUP BY contentobject_id, language_code, version, contentclassattribute_id ) t SET ezcontentobject_attribute.id = t.minid WHERE ezcontentobject_attribute.contentobject_id = t.contentobject_id AND ezcontentobject_attribute.language_code = t.language_code AND ezcontentobject_attribute.contentclassattribute_id = 183 I've opened a new issue -> https://jira.ez.no/browse/EZP-20737
          Hide
          gilles guirand added a comment -

          On a "memory limit overflow" issue, you should try "first" to optimize the PHP code on the memory layer,
          You moved the issue to "The total number of locks exceeds the lock table size", which is another interesting one, but not directly linked to the memory waste,

          A simple memory optimization is to always use asObjects=false inside a huge PHP loop,
          We did on this issue, it decreased a lot the memory usage (divide by 3), and work like a charm with 200k objects

          Show
          gilles guirand added a comment - On a "memory limit overflow" issue, you should try "first" to optimize the PHP code on the memory layer, You moved the issue to "The total number of locks exceeds the lock table size", which is another interesting one, but not directly linked to the memory waste, A simple memory optimization is to always use asObjects=false inside a huge PHP loop, We did on this issue, it decreased a lot the memory usage (divide by 3), and work like a charm with 200k objects
          Hide
          Yannick Roger (Inactive) added a comment -

          Hi Gilles,
          If you guys fixed some bugs or made some improvements, feel free to open a pull request on Github. We will be more than happy to merge it and you get the fame and glory in git log
          Cheers,
          Yannick

          Show
          Yannick Roger (Inactive) added a comment - Hi Gilles, If you guys fixed some bugs or made some improvements, feel free to open a pull request on Github. We will be more than happy to merge it and you get the fame and glory in git log Cheers, Yannick
          Hide
          Yannick Roger (Inactive) added a comment -

          Attached patches for ezpublish 4.5 ee + sp

          Show
          Yannick Roger (Inactive) added a comment - Attached patches for ezpublish 4.5 ee + sp
          Hide
          Gilles Ballini added a comment -

          Here is the Pull Request containing the patch Gilles Guirand's was talking about :
          https://github.com/ezsystems/ezscriptmonitor/pull/15

          Show
          Gilles Ballini added a comment - Here is the Pull Request containing the patch Gilles Guirand's was talking about : https://github.com/ezsystems/ezscriptmonitor/pull/15
          Hide
          André Rømcke added a comment -

          Additional improvement made for master:
          https://github.com/ezsystems/ezscriptmonitor/pull/15

          Show
          André Rømcke added a comment - Additional improvement made for master: https://github.com/ezsystems/ezscriptmonitor/pull/15

            People

            • Assignee:
              Unassigned
              Reporter:
              Nuno Oliveira (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 1 day, 3 hours, 42 minutes
                1d 3h 42m