Versioning of images

[bradwright]

Are there any plans to apply webassets versioning etc. to images as well? For example via a template tag and the CssRewrite filter?

[miracle2k]

Sure - but it's a relatively big untertaking, in particular in terms of API design.

• There would need to be a template tag, some utility function that can expire images included in HTML. This would have be work completely outside of the Bundle system.
• It would need to integrate with the manifest, so that versions can be served from memory.
• Some sort of filter would need to do in CSS files what what the template tag would do in HTML. The filter and the template tag must co-exist if the same file is referenced both ways.
• The complex part is really if the image files have to be rewritten to include the version in the filename. Just appending the version as a querystring should be straightforward.

What I would like to add as a first step, soon, is a datauri filter.

[tgecho]

I'll probably take a stab at this if no one else gets around to it. I'm doing a project without Django, so I'll need webassets to be able to do this soon.

• The basic idea doesn't seem too terribly complicated, but getting to work well will be a little tricky. It seems that cssrewrite/datauri contains many of the tools we'll need (finding/rewriting url()s, etc...).
• I'll need to use a hash in the new filename, so the manifest will be key. Also, maybe it would be wise to cache the hash using filename+timestamp as the key to avoid expensive hashing of images as much as possible?
• I'm not sure how to best handle files referenced through a html tag. Doing it on fly will mostly work (with an hit on first request), but precompiling is tricky.
  • I already don't use the template tag for bundles, mostly because of how messy I find it and the fact that my stuff is usually too dynamic for it to really work.
  • I think we need something that can take a directory of files and hash+copy+rename everything (matching a pattern?) into an output directory. Maybe a special bundle that takes a regex/replacement input and spits them into an output folder instead of combining?
• We need to be able to determine dependancies on the fly. Using depends= in sass bundles is already kind of annoying (though totally understandable), but there's no way I want to do that for any images I use. We'll already have a list of paths from the process of parsing out url()s... can we just append that to the bundle's depends or something?
• Summarized plan/wishlist
  1. CSS filter that rewrites urls for any files that exist locally (and thus can be renamed)
  2. Template tag that returns a rewritten url (or just the plain one in debug mode)
  3. Copy/rename found files to a specified output folder
  4. Cache the hash keyed on filename+timestamp
  5. Store filename => hashed filename in the manifest

Anyway, gotta run. That's my little brainstorm dump. Any other thoughts, hints, etc are welcome. I'll report back when I actually get started.

[miracle2k]

I didn't even think about depends=, but it's true - to implement auto_build, a changed image file that is referenced e.g. in CSS needs to be detected. Even without auto_build, such an external change would require the cache to be invalidated. Of course, it's always a possibility to just not support those things when image versioning is used, but it sure would be nice if it worked.

"Precompiling" images referenced via a tag could be done by parsing the templates, but that's generally a headache. I think that you're on the right track with a special bundle that does a hash->copy operation:

  ImagesBundle('images/**', output='generated/').build()

If this runs, then the hashed versions will be created, and the versions written to the manifest.

The template tag could internally just construct such a ImagesBundle and let it do it's work. If a manual build has already created a hashed version of an image, and the version is thus available in the manifest, the template tag would do nothing, and just use the existing version.

I.e. this would very nicely match exactly the process for normal bundles - you can either prebuild, or enable auto_build and have it happen upon request.

For images referenced in CSS files, a filter would be required, but it could work very much like the tag. Use the functionality already implemented in the ImagesBundle class to either generate a new hashed version of the image, or just return the filename of the one that was prebuilt.

When using an explicitly created ImagesBundle, the user obviously can specify a desired output directory. For the template tag and css filter, I don't think we can avoid introducing a new setting that says where the hashed images should be written to.

Also, it just occurs to me that this should not be just about images, but also, for example webfonts, so the name of that special bundle class would have to be different, maybe ExternalAssetsBundle.

Also, maybe it would be wise to cache the hash using filename+timestamp as the key to avoid expensive hashing of images as much as possible?

I'm not sure why that would be necessary. The manifest would supposedly already store the version (hash) of each file, which should be enough to determine whether an image has changed.

[tgecho]

I'm not sure why that would be necessary. The manifest would supposedly already store the version (hash) of each file, which should be enough to determine whether an image has changed.

I'm thinking that repeatedly hashing images to check for changes (especially in auto_build) would be an unfortunate waste of time/memory... but I could be missing something. I can understand avoiding premature optimization, so that can wait.

On renaming, do you think it's safe to just inject the hash into the filename before the extension, or should we expose a way to customize that behavior.

• Something like rename='%(name)s.%(version)s.%(ext)s'
• For external assets found via css filter, we could somehow base the rename behavior of the containing css bundle.

[miracle2k]

I'm thinking that repeatedly hashing images to check for changes (especially in auto_build) would be an unfortunate waste of time/memory..

Right, you'd want to compare the timestamps to see if there was a change. If the target filename contains a variable hash, that hash would be read from the manifest. Then, given that hash value, the target filename can be constructed and it's timestamp can be checked. This is how the process currently works if bundles with %(version)s placeholders, I think the same process should work fine for images - unless I am missing something.

On renaming, do you think it's safe to just inject the hash into the filename before the extension, or should we expose a way to customize that behavior.

Ideally, yes, I would like to see a way to customize it, but that certainly could be added later on. Rewriting the info from the bundle output might be a challenge, and wouldn't work for the template tag anyway, so I would go right ahead and make it a setting.

[miracle2k]

What if the low-level interface, for people who do not want to or cannot use a template tag (unsupported template language), looks like this:

>>> assets = ExternalAssets('img/*', 'fonts/*', output=''generated/')
>>> assets.url('fonts/Futura.eof')
'/media/generated/Futura.343ae1.eof'

You could easily pass the object to the template:

>>> setup_template_env(globals={'asset': assets.url})
>>> t('<img src="{{ asset("fonts/Futura.eof") }}">')

And a template tag implementation would merely have to transform:

 {% asset "fonts/Futura.eof" %}    
    ==>
 ExternalAssets('fonts/Futura.eof', output=env.config['ASSETS_OUTPUT_DIR']).url('fonts/Futura.eof')

[dstufft]

How are people handling image assets currently? I'm attempting to use webassets and I just realized all of my image files are broken because of this issue.

[fgimian]

Huge +1 to this request. I'm not aware of any solutions to this problem at the current time. In general, it would be great to see webassets handle any types of binary files, whereby one can request that webassets copy certain binary assets (e.g. images or fonts) to the static directory. track their MD5 checksum and append the appropriate GET parameter to ensure that old versions of the file are not served by any caches.

At line 445 in webassets/bundle.py, we have this code:

        try:
            try:
                final = filtertool.apply_func(filters_to_run, 'concat', [hunks])
            except MoreThanOneFilterError as e:
                raise BuildError(e)
            except NoFilters:
                final = merge([h for h, _ in hunks])

The first issue seems to be that webassets attempts to merge assets with a single item, so definitions such as the following fail:

assets.register('my_cat', 'images/cat.png', output='cat.png')

Because webassets attempts to open the file using a regular UTF-8 text mode.

Cheers
Fotis

[sp3c73r2038]

Huge +1 to this request, too! I would like to use external assets file such as webfonts, icon images.

[carljm]

This is not just a feature request, it is also a regression from earlier versions of webassets. In webassets 0.8, it was possible to set up a bundle containing a single binary file (e.g. an image) with no filters, and have webassets just copy it over and version-tag it. I just upgraded a project to webassets 0.10.1 and this is no longer possible, because FileHunk tries to decode everything it opens as UTF-8, so it crashes with UnicodeDecodeError on the binary file.

Even apart from all the grander feature ideas here (rewriting CSS links, etc), I would be quite happy just to have the level of functionality from webassets 0.8 back - the ability to handle binary files at all without crashing.

[miracle2k]

@carljm I just noticed that @qris has provided a pull request for adding binary support again, which I have missed so far, but seems to look good. You could give it a try:

#326