EISCAT 3D stakeholders meeting Dec 5th 2017

Time/Location

Tuesday December 5th 2017, Clarion Hotel conference centre. 13th floor, 09h15 to 16h00(-ish)

List of Attendees

Carl-Fredrik Enell (EISCAT)
Ingemar Häggström (EISCAT)
Börje Josefsson (SUNET)
John White (NeIC)
Tomasz Malkiewicz (NeIC)
Mattias Wadenstein (HPC2N)
Roy Dragseth (UiT)
Ari Lukkarinen (CSC)
Janne Ignatius (CSC)
Björn Torkelsson/Åke Sandgren (HPC2N)
Gunnar Bøe (UNINETT)
János Nagy (NSC, SE)
Vincent Garonne (UiO, NO)
Anders Tjulin (EISCAT)
Craig Heinselman (EISCAT)
Vidar Faltinsen
Helge Stranden
(as of 20/11/2017)

Objectives for the meeting

The E3DS stakeholder forum gathers representatives from the NeIC EISCAT_3D Support project, NeIC, EISCAT, EISCAT_3D users, national e-Infrastructure providers and network providers in order to achieve the goals:

1. To update each other about progress of projects and e-Infrastructure roadmaps;
2. To intensify the collaboration between the EISCAT_3D Support project and the providers for network and e-Infrastructures.

The general objective of this meeting is to bring together the EISCAT_3D technical community with national providers of HPC and networking from the relevant countries. The goals of the meeting include the need to update each other of statuses and time-lines of national providers and the EISCAT_3D activities.

In this meeting, the future proposed project between NeIC and the EISCAT_3D project will be presented for discussion. In this meeting we will have presentations from EISCAT_3D on the current plans and requirements for the future, based on the documents and updates that this project has produced. The future cooperation between the EISCAT_3D project potential e-infrastructure providers will be presented and discussed.

Agenda

• Meeting chair: Tomasz Malkiewicz

• 09h15 - 09h30 Welcome slides (Tomasz Malkiewicz, NeIC)

• 09h30 - 10h00 Status of the EISCAT_3D project slides (Craig Heinselman, EISCAT)

• 10h00 - 10h30 Coffee break

• 10h30 - 10h50 Nordic Network Plans slides (Börje Josefsson, SUNET)

• 10h50 - 11h00 Fibre cabling from Tromso to EISCAT slides (Helge Stranden, UNINETT)

• 11h00 - 11h30 Status of the E3DS project/Project Directive slides (John White, NeIC)

• 11h30 - 12h30 Lunch

• 12h30 - 13h00 Data formats, metadata, ENVRI slides (Carl-Fredrik Enell, EISCAT)

• 13h00 - 13h30 Data management for the LHC Experiments: ATLAS slides (Vincent Garonne, UiO, NeIC)

• 13h30 - 13h50 Progress on Norway National e-infrastructure slides (Roy Dragseth, UNINETT, UiT)

• 13h50 - 14h10 Progress on Swedish National e-infrastructure slides (Mattias Wadenstein, SNIC)

• 14h10 - 14h30 Progress on Finnish National e-infrastructure slides(Ari Lukkarinen ,CSC)

• 14h30 - 15h00 Coffee break

• 15h00 - 16h00 Conclusions

Meeting Notes

Meeting overview

Meeting chairperson, welcoming words, round of presentations

E3DS on track, three deliverables moved into new project.
Directive for new project written, plan to be prepared, effort doubled (2.25 FTE).

Round-table introductions...

Tomasz Malkiewicz (NeIC)
Vincent Garonne (UiO, NO)
Mattias Wadenstein (HPC2N)
Bjorn Torkelsson (HPC2N)
Janne Ignatius (CSC)
Ari Lukkarinen (CSC)
Janos Nagy (NSC, SE)
Anders Tjulin (EISCAT)
Carl-Fredrik Enell (EISCAT)
Vidar Faltinsen (UNINETT)
Borje Josefsson (SUNET) - left before lunch
Helge Stranden (UNINETT)
Ingemar Haggstrom (EISCAT)
John White (NeIC)
Roy Dragseth (UiT)
Gunnar Boe (UNINETT)
Craig Heinselman (EISCAT)

Welcome to two new people to the group Janos Nagy, Vincent Garonne

EISCAT_3D overview - Craig

Question: Why no ISR systems on the southern hemisphere?
Answer: Lack of infrastructure, difficult being power consumer on Antarctica

Question: Who operates the radar on Greenland?
Answer: NSF

Question: One container below each subarray? Size of the box
Answer: Yes, but they need to be smaller than for the PfP test array.

Comment: May have a lead or two for good programmers searching job (Mattias).

Question: Any more decisions on which experiments to run. Have effect on processing, bandwidth, etc
Answer: The plan is to have standard experiments to start with. The subarray computing will not change significantly over time. Will build for flexibility.

Question: Storage needs on day one?
Answer: On the order of 2 PB new data for long term storage per year. Store as much as we can afford. Start off simple and expand as we can. 500 TFLOPS is not required all the time. We look at a 30y lifetime.

Question: Timeline for experiments on the site?
Answer: E3D opens up for experiments around end of 2021, after the commissioning. Scheduling of experiments one month before the experiment date. 2022 onwards fully operational.

Question: Timeline for in-kind contribution from Japan?
Answer: This month or latest by mid-January next year. Good thing is NIPR owns the design for the amplifiers, can use another manufacturer than Mitsubishi.

Question: How are measurements requested? Answer: Month by month basis... we schedule to the next month. Currently Ingemar does this.

Nordic Network Plans

- Nordic NRENs support for E3D

- Ring topology suggested. Planning for a fibre ring following the roads E10 - E45 - E8 - E6 - E10 introduces redundancy.
- Need new connections Tromso-Skibotn-Kilpis, Kilpis-Karesuvanto, Karesuando-Svappavaara.
- This custom build comes with a cost. Decision on OC location is important wrt data flow.
- Suggested: bandwidth as a service, provided to EISCAT, one contact point (Nordunet), monthly billing.
- Simulation of data flow should be done on the real network.
- No existing HPC/HTC centres can handle the data rates yet.
- Integration with NT1 is important for costs (coordination of staff).

Comment: Clarifies that the NT1 is only for the operations/on-call NOT to distribute throughout the whole nordics. Interaction with tier one computing means organisation of staff. There will only be one computing centre. (Mattias)

Question: For the e-infra are there any other sites considered outside the very Northern map? Answer: FI/SE/NO: confirm that the northern sites are the only candidates. Nothing further south.

Link from Tromso to the Skibotn

- Sub-sea cable will be ready Feb 2018.
- Landings ready in December.
- Connections from powerline pole 130 to EISCAT 3D site and Skibotn field station will be put in tubes. Ready summer 2018.
- Internet access from UiT research station from early summer 2018.

Status of the E3D support project

Comment: Modern computer does not consume much power when idle. So switching off maybe not the best idea.
Question: What are the expected service hours? When is availability needed? Durability requirements?
Answer: More forgiving than LHC. Loss of small amounts of data normally not a big issue, unless coordinated experiments with rockets. Operation in general continuous but often with low cadence. It is a research observatory, thus observations over all hours. Short outages not problematic.
Question: What are the computing centres going to operate? Clarification needed before discussing organisation.
Answer: EISCAT also needs to know costs for different computing solutions. EISCAT is limited by funding.
Question: Service level is important. Reliability and accessibility, vs. cost. (Ari)
Answer: EISCAT does not expect 24/7 reliability.

Data requirements for E3D

Explains the difference between the current and the E3D case. (100 \times 2 \times heights)
, Data rate, data sorting, formating granularity, catalogue search
ENVRI (reference model) OIL-E (Open Information Linking for Environmental science)

Question: Mirror for archive? (Helge)
Answer: The archive will be mirrored, not the ring buffer, or the 20 PB buffer

Q: Why funding organisation requirement for authentication and access?
A: There are limits in the statutes on who can use the data, plus an embargo period. The EISCAT Blue book determines the rules and the data is open to whole association after embargo.

C: "Long term storage" is a better term than "archive". Long term storage, consider hot vs cold storage. Better to avoid confusion.

C: For environmental monitoring, check NAV, a network administration system used successfully in Tromso (nav.sigma2.no)

Question: Is the Open Data FAIR? (??)

Comment: For feedback from users? What do users want from us? (Anders)
Comment: Feedback from users is the point of Scientific Advisory Committee (SAC). (Craig)

Data management for LHC experiments Atlas, Rucio & Friends

- Presentation of Rucio, the ATLAS data management system.
- Handling all data products in a heterogeneous distributed environment.
- Handles 40 PB per month, 40 M files, 335 PB data at 130 sites.
- Datasets comparable to Dirac: sites operate only storage.
- Mostly Python software.
- Modules, REST API.
- Predicted exabyte scale during LHC run 4.
- Code development on Github, 5-6 FTEs, 40 contributors.

Comment: Workshop at CERN March 1-2, at least John should participate.
Comment: Looks promising, would facilitate mirroring to Japan and UK.

Question: What is the relation to EOS?
Answer: Rucio is a service that runs on top of a file management system, can be EOS, dCache etc.

Question: Maximum data to one data centre? (Gunnar)
Answer: CERN will see about 10% of the total transfers. (Mattias)
Question: What system did you have before? (Tomasz)
Answer: It was called Don Quixote.
Question: What does it mean when you delete "unpopular" data? Is there a copy at tier-0? (Gunnar)
Answer: Only secondary copies are deleted.
Question: Search engine? Extracts of data search using specific criteria. (Roy)
Answer: Metadata search is implemented, at least the functionality is there for ATLAS.
Question: How many peope are involved? Any correspondence to the amount of data?
Answer: To run the service, 2 FTE plus developers
Comment: Transferring between sites could lead to transferring errors. There is a need to look out.
Question: Interest from other LHC experiments?
Answer: Yes, CMS is evaluating. We need to consider the amount of data (LHCb). DIRAC (WMS) used by LHCb migth be integrated with Rucio for the BELLE-II use case.
Question: Do you see any potential for the EISCAT community?
Answer: Could identify collaborational aspects, in particular sharing experiences.

Progress on Norway National e-infrastructure

- NREN Uninett, Subsidiary company Uninett Sigma 2
- Current HPC systems: Abel (Oslo), Stallo (old UIT), Fram (new system UiT, 1000 nodes, 30000 cores).
- Data transferred from Oslo to Troms?? (with 2-site GPFS).
- Planning facilitating use through a common architecture: App engine and store, development through Docker images, Kybernetes API to backend systems.
- Future: 2 main systems, 2 year upgrade cycle, locations open. Next upgrade (2018) is a big cluster for NTNU.
- Data storage: NIRD expansion by 10 PB until 2019.
- NAV developed by UNINETT, Open Source.

Questions: Vidar: How about the LHC research at UiB and UiO? (Vidar)
Answer: Not decided since this requires large RAM computing rather than HPC. Memory requirements also. There are complicated storage requirements. The process is under discussion. (Gunnar)

Question: What domain are the EISCAT users in, in terms of authorisation mechanisms?
Question: User community of EISCAT? Edugain? How to deal with Asain users? Determines Auth mechansims. (Gunnar)
Answer: For UNINETT... there is FEIDE-2. Based on SAML.

Question: How big will the B1 be? (Janne)
Answer: Bigger than Fram by lots.

Progress on Swedish National e-infrastructure

- SNIC 2.0 starts 2018. New organization: consortium of 10 universities, cofunding 50 % VR 50 % universities, new cost model based on usage instead of hosting (beneficial for UmU). Board with strategic focus.
- HPC2N, UmU: New tape library was installed in October.
- Waiting for more GPU nodes for Kebnekaise cluster.
- Monitoring with Ganglia.
- NSC, LiU: Procurement in progress for new big cluster (architecture open) and tape library.

Question: How do you find the reliability rate of your tape system? (Gunnar)
Answer: The old one is OK, new one is also OK. There is no tape drive error yet.

Question: What about the cost for hosting facility? (Gunnar)
Answer: Hosting facilities fully paid.

Comment: Research Infra computing are paid by RI projects... e.g. EISCAT would be an EISCAT RI project.
Comment: NeIC is not included... this is paid by Swedish research council.
Comment: For Kebnekaise we aim for 20/30 GPU nodes? Aiming for better than K80 (Bjorn).
Comment: We currently have 36 in Kebnekaise... high pressure in applications for them.

Progress on Finnish National e-infrastructure

- Funet 2020 upgrade with new denser fibre topology (need EISCAT 3D network design decision in near future).
- New hardware procurements in progress 2018-2021, idea is one contract with one vendor, capability computing and performance computing systems.
- Data management solution for end users and projects, also procurement.
- Storage solutions in use: Parallel filesystems (Lustre, GPFS etc), object storage (Ceph), generic medium term, long term (OAIS).
- Cloud services.
- EU projects: EUDAT H2020 ending, EOSC Hub, future ENVRIPLUS EOSC.

Question: Budget of 33m Euros... from where? (Vidar)
Answer: Directly from the ministry of education/culture. CSC owns the systems.
Comment: CSC is growing rapidly. Increase workforce up to 330-340 soon. Expect to hire people. Even data scientists to Kajaani these days.

Conclusions

The conclusions drawn from this meeting (according to the E3DS project manager). Comments welcome!