How can I use Python to obtain the relationship between the time and gap of each log print during the solving process of Scip solver

[BloomAndDeath]

I just came into contact with the SCIP solver. I found that the SCIP solver may be blocking all processes

The proof is as follows:

1. I used the SCIP solver to solve the instance in the main thread, and opened another thread to obtain the gap of SCIP every 1 second, which can only obtain the gap value of the last time;
2. I placed the SCIP solver in one thread and obtained the gap of SCIP every 1 second in another thread. I opened two threads in the main thread, but the result was that only the last gap value could be obtained.

I want to obtain the relationship between time and gap as recorded in the log of the solver during the solving process. Using a handler with event type BESTSOLFOUND can only retrieve a portion of the time and gap values. If I want to obtain all the time and gap data like the log output, how can I do it? If anyone can help solve it, I would be extremely grateful

The log information recorded during the execution process is as follows：

 time | node  | left  |LP iter|LP it/n|mem/heur|mdpt |vars |cons |rows |cuts |sepa|confs|strbr|  dualbound   | primalbound  |  gap   | compl. 
t 0.0s|     1 |     0 |     0 |     - | trivial|   0 | 412 | 176 |   0 |   0 |  0 |   0 |   0 | 1.000156e+05 | 2.460007e+03 |3965.66%| unknown
p 0.0s|     1 |     0 |     0 |     - |  clique|   0 | 412 | 176 | 176 |   0 |  0 |   0 |   0 | 1.000156e+05 | 6.292663e+03 |1489.40%| unknown
p 0.0s|     1 |     0 |     8 |     - | vbounds|   0 | 412 | 176 | 176 |   0 |  0 |   0 |   0 | 1.000156e+05 | 6.341714e+03 |1477.11%| unknown
  0.0s|     1 |     0 |   851 |     - |  9350k |   0 | 412 | 176 | 176 |   0 |  0 |   0 |   0 | 7.746556e+03 | 6.341714e+03 |  22.15%| unknown
  0.1s|     1 |     0 |   879 |     - |    19M |   0 | 412 | 176 | 178 |   2 |  1 |   0 |   0 | 7.731929e+03 | 6.341714e+03 |  21.92%| unknown
  0.1s|     1 |     0 |   910 |     - |    29M |   0 | 412 | 176 | 183 |   7 |  2 |   0 |   0 | 7.714161e+03 | 6.341714e+03 |  21.64%| unknown
  0.1s|     1 |     0 |   948 |     - |    39M |   0 | 412 | 176 | 185 |   9 |  3 |   0 |   0 | 7.700883e+03 | 6.341714e+03 |  21.43%| unknown
  0.2s|     1 |     0 |   995 |     - |    44M |   0 | 412 | 176 | 186 |  10 |  4 |   0 |   0 | 7.693293e+03 | 6.341714e+03 |  21.31%| unknown
  0.2s|     1 |     0 |  1013 |     - |    49M |   0 | 412 | 176 | 187 |  11 |  5 |   0 |   0 | 7.690795e+03 | 6.341714e+03 |  21.27%| unknown
  0.2s|     1 |     0 |  1035 |     - |    57M |   0 | 412 | 176 | 189 |  13 |  6 |   0 |   0 | 7.688735e+03 | 6.341714e+03 |  21.24%| unknown
  0.3s|     1 |     0 |  1058 |     - |    60M |   0 | 412 | 176 | 191 |  15 |  7 |   0 |   0 | 7.687740e+03 | 6.341714e+03 |  21.22%| unknown
  0.3s|     1 |     0 |  1085 |     - |    66M |   0 | 412 | 176 | 193 |  17 |  8 |   0 |   0 | 7.686455e+03 | 6.341714e+03 |  21.20%| unknown
  0.3s|     1 |     0 |  1109 |     - |    69M |   0 | 412 | 176 | 197 |  21 |  9 |   0 |   0 | 7.685850e+03 | 6.341714e+03 |  21.20%| unknown
  0.4s|     1 |     0 |  1144 |     - |    78M |   0 | 412 | 176 | 199 |  23 | 10 |   0 |   0 | 7.684444e+03 | 6.341714e+03 |  21.17%| unknown
  0.4s|     1 |     0 |  1165 |     - |    78M |   0 | 412 | 176 | 200 |  24 | 11 |   0 |   0 | 7.683881e+03 | 6.341714e+03 |  21.16%| unknown

[Joao-Dionisio]

I don't know if there's a super nice way to get what you want. You can get more information than when using BESTSOLFOUND if you use LPEVENT, which consists of solving a node's initial LP and completely solving the LP. It still wouldn't give you everything, though, because intermediate solutions wouldn't trigger the event handler.

You can use a workaround, however. Suppose you're minimizing f(x). Then you can add an auxiliary variable z, add the constraint f(x) <= z, and change your objective to minimizing z. This reformulation is equivalent to the original problem, but now you can try to catch the event GLBCHANGED. I'm not 100% sure, but I'm hoping that the event will activate every time the global lower bound of variable z increases, which corresponds to the dual bound increasing. This, in conjunction with the BESTSOLFOUND event, should activate every time the gap changes.

EDIT: If I may ask, how did you come in contact with SCIP? To help us understand where our users are coming from 😃

[BloomAndDeath]

hi @Joao-Dionisio ，Thank you for your response. Do you know under what circumstances these intermediate log records are recorded by Scip?
I have been working in the field of mixed integer linear programming recently and found a SCIP solver while reading papers

[Joao-Dionisio]

While solving an LP, I think SCIP doesn't have access to the intermediate solutions, it's Soplex (an outside LP-solver) that's doing it by default. However, in each node, the LP is solved multiple times because there are separation rounds that add cuts to strengthen the LP-relaxation. And it's these in-between LPs that SCIP doesn't have an event for, but records the progress.

[BloomAndDeath]

@Joao-Dionisio Thank you again for your response. I may not have wanted to obtain the intermediate solution. The progress recorded by SCIP includes time and gap parameters. Can I read these two data in Python when SCIP records them

[Joao-Dionisio]

If you don't want the intermediate solutions, then using an event handler with the LP-EVENT and BESTSOLFOUND events will be enough.

Accessing the data directly is not possible with PySCIPOpt.

[Joao-Dionisio]

We added a new event type to SCIP SCIP_EVENTTYPE_DUALBOUNDIMPROVED, which will be brought over to PySCIPOpt after the next SCIP release.

EDIT: Also SCIP_EVENTTYPE_GAPUPDATED, which catches changes on the dual and the primal side.