DQCS Shift Notes and Checklist ============================== Documentation ------------- * latest `training `_ * `Shift DataBase `_ * `Bicocca Docs `_ * `Operation Meeting `_ setup `instructions `_ and `CodiMD `_ * `Simulation Meeting `_ setup `instructions `_ and `CodiMD `_ Prerequisites ------------- - ☐ ELog access - ☐ Subscribed to `lhcb-simulation`, `lhcb-distributed-analysis`, `lhcb-geoc` mailing lists - ☐ Join `Simulation` and `Computing Operations` Mattermost Channels - ☐ Get Grid Certificate - ☐ `lhcb_shifter` role available on DIRAC portal - ☐ test generation of DQCS report made from `this `_ or `this `_ repo MC Quality checks ----------------- Nightly Tests `docs `_ Gauss LHCbPR Tests `docs `_ Dirac portal `docs `_ Every couple of hours ~~~~~~~~~~~~~~~~~~~~~ - ☐ Monitor `Dirac Portal `_ plots for anomalies and report it on `Operation channel `_ monitoring the plots on desktops (Concezio Bozzi's ones are OK). They appear going to "Settings->Group : lhcb_shifter". Then go to "Application->Administration->Public State Manager", select "Desktops->Shared Desktops" and choose the desktop with saved the plots to inspect. Report in Computing Operations Once a Day ~~~~~~~~~~ - ☐ Visit `LHCb Nightly builds reports `_ and update Simulation Data Quality `codimd `_ - ☐ lhcb-sim10 - ☐ lhcb-sim10-dev - ☐ lhcb-sim11 - ☐ Visit `LHCbPR Simulation dashboard `_ for the following checks: - ☐ Gauss software performance check - ☐ Detailed check to commissioning future simulation campaign Once a Week ~~~~~~~~~~~ - ☐ Check `LHCb Nightly builds reports `_ - ☐ lhcb-sim11-g4 - ☐ lhcb-sim09-cmake - ☐ Visit `LHCbPR Simulation dashboard `_ for the following checks: - ☐ Run 2 detailed simulation check - ☐ Run 3 detailed simulation check For each slot, check: ~~~~~~~~~~~~~~~~~~~~~ * the given slot builds for all platforms * all merge requests were successfully merged in the given nightly slot * if there is an arrow next to the project name, click on the blue page icon and verify it in checkout log * there are no new test failures that appeared for the x86_64_v2-centos7-gcc11-opt platform * compare to previous build (or few builds) using the [Compare with previous builds] button * compare the latest lhcb-sim10-dev slot with the latest lhcb-sim10 slot using the [Compare slots] button .. warning:: `lhcb-sim11` has more platform that needs to be monitored: - `x86_64_v2-centos7-gcc11+detdesc-opt`, - `x86_64_v2-centos7-gcc12-opt`, - `x86_64_v2-centos7-gcc12+detdesc-opt` - `x86_64_v2-el9-gcc12-opt` .. note:: The `*-dbg` tags are less important, especially as far as tests are concerned. Week schedule ============= Friday before ------------- - ☐ `time=11:30` Attend `LHCb Computing Operations `_ meeting to acknowledge the status of the system Monday ------ - ☐ `time=10:00` copy and update `LHCb Computing Operations `_ meeting event and chair it asking news about the tiers and other topics - ☐ do Once a day tasks - ☐ check nightly report prepared by the previous shifter and update it using `this repo `_ and update Simulation Data Quality codimd Tuesday ------- - ☐ do Once a day tasks - ☐ `time=16:30` attend `Simulation Meeting `_ reporting information on previous week and upload a `.md` and a `.html` copy of this `codimd `_ - ☐ clear out tables on codimd and start preparing the report for the next shifter Wednesday --------- - ☐ `time=10:00` copy and update `LHCb Computing Operations `_ meeting event and chair it asking news about the tiers and other topics - ☐ do Once a day tasks Thursday -------- - ☐ do Once a day tasks Friday ------ - ☐ `time=10:00` copy and update `LHCb Computing Operations `_ meeting event and chair it asking news about the tiers and other topics - ☐ do Once a week tasks - ☐ do Once a day tasks Saturday -------- - ☐ do Once a day tasks Sunday ------ - ☐ do Once a day tasks - ☐ generate nightly report using `this repo `_ and update the `codimd `_ for the next user