Asset download during nextjs export

[sandys]

Feature request

A related conversation to this feature happened here #9524 (comment)

if i use a headless CMS (even wordpress or graphcms or whatever), the asset url is used in the static html.

There are two preferences here - that asset links be used as such.
But more likely - download the asset, build the html (link to it locally) and then layer a CDN in front. This is the far more acceptable practice.

This also ties in very well with using deployment systems like Netlify - which have a far, far more suitable globally available infrastructure than something like DatoCMS or Graphcms. So if im using Netlify as deployment, i would want everything to be served from Netlify domain and let it work its magic.

We understand that this increases build times. This is ok for our usecase where we use a CMS just for the sake of usability, but deployment has to follow guidelines of compliance (geographic restriction of assets, etc) so we would like the entire build to be inside the export folder. This would mean that CMS that have CDN in front of it (e.g. datocms and www.datocms-assets.com which is Imgix) may not be ideal.

There are tons of similar requests.

https://spectrum.chat/next-js/general/how-would-you-handle-importing-remote-images-on-nextjs-static-export~30b2ba84-bc27-4da7-9ec8-21e4d5d287a3

on gatsby side as well - gatsbyjs/gatsby#14076

https://spectrum.chat/gatsby-js/general/adding-remote-images-during-node-creation~e704e6fb-24b2-46c6-b1fc-93189d2e28a4

https://github.com/njosefbeck/gatsby-source-stripe/#downloading-files

[Timer]

This is covered by #9524

[sandys]

@Timer sorry for the late response. but this is NOT covered by #9524 . In fact, i was told there to open a new issue since this will not be considered for inclusion under SSG improvements.

Is it possible to do this using any kind of custom code, etc ? this is the only thing blocking us from moving entirely to Nextjs from Gatsby.

[erhankaradeniz]

@Timer I think @sandys is right here, I'm not seeing anything in that RFC which would solve this issue? Please correct me if I'm wrong, since I'd like to make use of this feature too since I switched to almost 100% SSG pages with the canary builds.

[sandys]

hi guys,
now that nextjs 9.3 is released with next-gen SSG support ( https://nextjs.org/blog/next-9-3#next-gen-static-site-generation-ssg-support ), im wondering if there is any way to achieve what was mentioned in this bug ?

[sandys]

@Timer - any chance you can take a look at this and comment ? if there's a way to do this using the existing Nextjs framework, that would be awesome as well.

[franciscorubin]

Any news on this? Ideally this should just store images fetched during getStaticProps on .next/static/images like the rest of the images of the website. The fetching will only happen on build time (getStaticProps), so this shouldn't be a problem technically.

[mixtur]

I think it should also avoid duplicate downloads and allow post processing of whatever was downloaded afterwards.
So you mark something as an asset in getStaticProps and then in another magic export you do something with it. (Resize, compress etc)

[zommerberg]

Any news on this?

[Ecksters]

In my case my headless CMS is internal-only, and only accessible during the build process, currently I'm having to hack together a solution to copy the uploads folder from the headless CMS into my build export's out folder so that the URLs provided by the CMS can be accurate.

It'd be awesome, if NextJS could handle fetching these assets during the build for me, including potentially doing the image optimizations that are already available during the build.

[sandys]

Same issue for us. The CMS is private. We would love this feature.

…

On Thu, 11 Feb, 2021, 11:49 Daniel Eck, ***@***.***> wrote: In my case my headless CMS is internal-only, and only accessible during the build process, currently I'm having to hack together a solution to copy the uploads folder from the headless CMS into my build export's out folder so that the URLs provided by the CMS can be accurate. It'd be awesome, if NextJS could handle fetching these assets during the build for me, including potentially doing the image optimizations that are already available during the build. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10147 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAASYUZGXQAMGSJI57D7OWLS6NZIRANCNFSM4KIQRIBA> .

[le-martre]

This would really be a superb feature, I really need it as well :) Will check out if it's possible to hack something together but I imagine someone smarter than me would have figured it out by now :p

[sandys]

Unfortunately we couldn't figure it out. We had to move one of our projects to Gatsby cos of this :( Hoping that nextjs includes this.

…

On Fri, 16 Jul, 2021, 15:34 Martin Vandersteen, ***@***.***> wrote: This would really be a superb feature, I really need it as well :) Will check out if it's possible to hack something together but I imagine someone smarter than me would have figured it out by now :p — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22057 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAASYUZBJYL6LLATT2RP5YTTX77Z5ANCNFSM5APIR4HQ> .

[le-martre]

I actually managed to hack something together. Beware, it's as dirty as it gets, I just stopped when I got it working but didn't clean anything up and on top of that I suck at JS..

Basically, I just added a step after "next build && next export" that parses all the html files and finds all the images' srcs and srcsets. I filter them down to only those that start with specific strings (example "https://www.datocms-assets.com/"), I download everything then replace everything in the HTML and JSON files.

It did work on Netlify as well first try.. Of course I STRONGLY encourage anybody that wants to use it to go over everything and refactor it because it really is just a Proof of Concept. But I figured it might help other people :) Cheers !

https://gist.github.com/MartinVandersteen/7235dbf42f601d83744067a72c4348fa

[robomatic]

only slightly less dirty ... I riffed on yours @MartinVandersteen.
https://gist.github.com/robomatic/8dccf289e33aac6ef92a1c4b6e69ca20

[le-martre]

Works really well ! Thanks a lot for that.

For anyone using Datocms as well be careful that they seem to have appended the project number at the end of the URLS, so the url to replace might have become https://www.datocms-assets.com/12345 for some of you (12345 being a project id)

[le-martre]

I had to update yours again so that it changes URLs in the JSON files as well. Not changing them caused weird issues where on the first load of some pages it would grab the datocms URL from the JSON file but would then use the local url on subsequent reloads.

https://gist.github.com/MartinVandersteen/7235dbf42f601d83744067a72c4348fa

[jens1101]

I recently had this same challenge and stumbled upon this discussion. Our private CMS returns to us URLs for each asset and we wanted to make these external assets form part of the NextJS build. I took a different approach from the suggestions listed here, so I decided that I wanted to share my solution.

Step 1: Add the following to next.config.js:

module.exports = {
  serverRuntimeConfig: {
    projectRoot: __dirname
  }
  // ...
};

This allows us to easily get the directory path of the repo later on.

Step 2 In the getStaticProps for each page we fetch and parse the necessary data from the CMS. During the data parsing process we also have to parse each asset. As part of that step we download each asset to the public directory. I decided to download them to "public/uploads" for organisational purposes. This is also where the server runtime config comes in handy. The code looks something like this:

const targetDirectory = path.join(
  serverRuntimeConfig.projectRoot,
  'public/uploads',
);

await downloadAsset({
  url,
  targetDirectory,
}),

The download function is fairly similar to this SO answer.

Step 3: Use the new path to the asset in the public directory. This is fairly simple since "/" refers to the public directory. Here's my code:

return `/uploads/${path.basename(url)}`;

And that's it. Now the assets will be part of your build, since they are inside of the "public" folder.

(Recommended) Step 4: I recommend you add the asset target path to your .gitignore file so that the assets don't accidentally get checked into source control.