Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ID-1315] Download list of Azure IP addresses #1736

Draft
wants to merge 2 commits into
base: develop
Choose a base branch
from
Draft

[ID-1315] Download list of Azure IP addresses #1736

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0cd70fc
Download and parse Azure IP addresses
samanehsan Jul 15, 2024
f1a73af
Merge branch 'develop' of github.com:DataBiosphere/jade-data-repo int…
samanehsan Jul 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions src/main/java/bio/terra/app/configuration/ApplicationConfiguration.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import bio.terra.service.resourcemanagement.azure.AzureResourceConfiguration;
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.MapperFeature;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.datatype.jdk8.Jdk8Module;
Expand All @@ -11,8 +12,11 @@
import com.google.auth.oauth2.ServiceAccountCredentials;
import com.microsoft.sqlserver.jdbc.SQLServerDataSource;
import java.io.IOException;
import java.net.URL;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
Expand Down Expand Up @@ -444,4 +448,29 @@ public WebServerFactoryCustomizer<TomcatServletWebServerFactory> tomcatCustomize
connector ->
connector.setEncodedSolidusHandling(EncodedSolidusHandling.DECODE.getValue()));
}

@Bean("azureIPs")
public Map<Integer, List<String>> azureIPs() throws IOException {
URL url =
new URL(
"https://download.microsoft.com/download/7/1/D/71D86715-5596-4529-9B13-DA13A5DE5B63/ServiceTags_Public_20240708.json");
Copy link
Contributor Author

@samanehsan samanehsan Jul 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just hard-coded this url for testing btw. It actually changes when the file is updated once a week so our options are to:

  1. Parse the download url from https://www.microsoft.com/en-us/download/details.aspx?id=56519
  2. Use the https://learn.microsoft.com/en-us/rest/api/virtualnetwork/service-tags/list?view=rest-virtualnetwork-2024-01-01 API endpoint (this requires a subscription id though)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah dang yeah thats annoying, i think using the API endpoint is the move, not sure about the subscription we should use for that though, id ask about it in #dsp-azure

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The second object won't get use Azure IP ranges. It'll just list the Public IPs within a subscription, which are a specific Azure resource

ObjectMapper objectMapper = new ObjectMapper();
JsonNode data = objectMapper.readTree(url);
JsonNode values = data.get("values");
// Map {Region ID: List of IP addresses}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm mapping based on the region id here, since we may need to only allow IPs from certain regions for AXIN. I'm still trying to find the regions those ids correspond to...

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like you can maybe use this endpoint to list the regions? https://learn.microsoft.com/en-us/rest/api/resources/subscriptions/list-locations?view=rest-resources-2022-12-01&tabs=HTTP

HashMap<Integer, List<String>> azureIPs = new HashMap<>();

for (JsonNode v : values) {
JsonNode properties = v.get("properties");
Integer regionId = properties.get("regionId").asInt();
ArrayList<String> addressPrefixes =
objectMapper.convertValue(properties.get("addressPrefixes"), ArrayList.class);
if (!azureIPs.containsKey(regionId)) {
azureIPs.put(regionId, addressPrefixes);
} else {
azureIPs.get(regionId).addAll(addressPrefixes);
}
}
return azureIPs;
}
}
5 changes: 4 additions & 1 deletion src/main/java/bio/terra/service/filedata/DrsService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@ public class DrsService {
private final DrsDao drsDao;
private final DrsMetricsService drsMetricsService;
private final AsyncTaskExecutor executor;
private final Map<Integer, List<String>> azureIps;

private final Map<UUID, SnapshotProject> snapshotProjectsCache =
Collections.synchronizedMap(new PassiveExpiringMap<>(15, TimeUnit.MINUTES));
Expand All @@ -133,7 +134,8 @@ public DrsService(
EcmConfiguration ecmConfiguration,
DrsDao drsDao,
DrsMetricsService drsMetricsService,
@Qualifier("drsResolutionThreadpool") AsyncTaskExecutor executor) {
@Qualifier("drsResolutionThreadpool") AsyncTaskExecutor executor,
@Qualifier("azureIPs") Map<Integer, List<String>> azureIPs) {
this.snapshotService = snapshotService;
this.fileService = fileService;
this.drsIdService = drsIdService;
Expand All @@ -148,6 +150,7 @@ public DrsService(
this.drsDao = drsDao;
this.drsMetricsService = drsMetricsService;
this.executor = executor;
this.azureIps = azureIPs;
}

private class DrsRequestResource implements AutoCloseable {
Expand Down
3 changes: 2 additions & 1 deletion src/test/java/bio/terra/service/filedata/DrsServiceTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,8 @@ void before() throws Exception {
ecmConfiguration,
drsDao,
drsMetricsService,
new SimpleAsyncTaskExecutor()));
new SimpleAsyncTaskExecutor(),
Map.of()));
when(jobService.getActivePodCount()).thenReturn(1);
when(drsConfiguration.maxDrsLookups()).thenReturn(1);

Expand Down
Loading